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ABSTRACT 

This  two  volume  final  report  summarizes  the  analysis,  design  and  development 
activities  associated  with  the  Educational  Technology  Assessment  Model  (ETAM). 

It  contains  relevant  background  information  and  results  of  prior  studies  lead- 
ing to  the  finalized  ETAM  procedures  and  computerized  routines.  A comparison  of 
the  manual  versus  the  automated  approach  is  included.  Data  base  structures  and 
the  ETAM  program  flow  are  described  and  related  to  each  other.  The  appendix 
includes  the  results  of  a study  on  the  indexing  of  innovations  and  the  assign- 
ment of  taxonomic  descriptors  to  courses,  job/tasks,  and  instructional  vehicles. 
The  appendix  also  includes  program  documentation  on  the  ETAM  Range-of-Effect, 
bibliographic  references,  and  additional  information  supporting  the  ETAM  design. 
The  detailed  ETAM  procedures  supporting  this  report  are  included  in  TAEG  Report 
No.  12-3,  Phase  1 1- A Report.  TAEG  Report  No.  32,  The  Development  of  Scaling 
Procedures,  supports  the  computerized  approach  taken  in  scaling  variables. 

Other  important  background  and  reference  information  can  also  be  found  in  a 
report  written  by  Drs.  R.  B.  Miller  and  A.  F.  Smode  titled  "Major  Innovations 
in  Training  Technology." 

Phases  I,  II,  1 1- A,  III  and  IV  were  accomplished  by  the  IBM  Federal  Systems 
Division  with  the  Training  Analysis  and  Evaluation  Group,  Orlando,  Florida, 
providing  technical  guidance  and  support.  The  overall  DOTS  objective  is  to 
provide  Naval  Education  and  Training  Command  (NAVEDTRACOM)  management  with 
additional  tools  in  the  form  of  computerized  mathematical  models  to  assist 
in  predicting  the  quantitative  impact  of  training  resource  decisions.  The 
planning  process  will  be  enhanced  by  providing  decision  makers  with  the 
capability  to  economically  and  rapidly  consider  a wider  range  of  alternatives. 

Phase  I was  a study  and  definition  effort  resulting  in  a complete  functional 
description  of  the  NAVEDTRACOM;  a strategic  definition  of  the  social,  political, 
economic  and  technological  environments  pertinent  to  the  naval  education  and 
training  system  in  the  1980's;  a list  of  existing  and  potential  models  amenable 
to  computerization  and  to  improving  the  decision-making  process. 

Phase  II  was  devoted  to  the  selection  and  development  of  three  mathematical 
models  from  the  Phase  I list  of  candidates.  The  three  were  the  System  Capabili- 
ties/Requirements and  Resources  (SCRR),  the  Individualized  Training  Simulation 
System  (ITSS),  and  the  Training  Process  Flow  (TPF)  models. 

Phase  III  centered  on  evaluating  the  selected  models  at  the  Fleet  Training 
Center,  Norfolk,  VA.  An  important  recommendation  from  the  Test  and  Evaluation 
conducted  during  Phase  III  was  that  DOTS  should  investigate  model  applications 
at  higher  command  levels. 

Phase  IV  responded  to  the  recommendations  of  Phase  III  by  (1)  operating  and 
testing  at  TRAPAC,  San  Diego,  CA,  the  models  developed  in  Phase  II,  and  (2) 
developing  a new  Training  Requirements  Analysis  Model  (TRAM)  and  field  testing 
it  at  CNTECHTRA,  Memphis,  Tenn. 

The  major  effort  supporting  Phase  II-B  was  the  FTAM  procedural  development 
during  Phase  1 1 - A . In  addition  to  the  procedural  development,  computer 
applications  were  described,  scenarios  using  the  ETAM  procedures  were  pre- 
pared, and  ETAM  validity  and  logical  structure  were  confirmed. 
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Phases  I,  II,  II-A,  III  and  IV  were  accomplished  by  the  IBM  Federal  Systems 
Division  with  the  Training  Analysis  and  Evaluation  Group,  Orlando,  Florida, 
providing  technical  guidance  and  support.  The  overall  DOTS /objective  is  to 
provide  Naval  Education  and  Training  Command  (NAVEDTRACOM)  management  with 
additional  tools  in  the  form  of  computerized  mathematical  models  to  assist 
in  predicting  the  quantitative  impact  of  training  resource  decisions.  The 
planning  process  will  be  enhanced  by  providing  decision  makers  with  the 
capability  to  economically  and  rapidly  consider  a wider  range  of  alternatives. 

Phase  I was  a study  and  definition  effort  resulting  in  a complete  functional 
description  of  the  NAVEDTRACOM;  a strategic  definition  of  the  social,  political, 
economic  and  technological  environments  pertinent  to  the  naval  education  and 
training  system  in  the  1 980 * s ; a list  of  existing  and  potential  models  amenable 
to  computerization  and  to  improving  the  decision-making  process. 

Phase  II  was  devoted  to  the  selection  and  development  of  three  mathematical 
models  from  the  Phase  I list  of  candidates.  The  three  were  the  System  Capabili- 
ties/Reauirements  and  Resources  (SCRR),  the  Individualized  Training  Simulation 
System  (ITSS),  and  the  Training  Process  Flow  (TPF)  models. 

Phase  III  centered  on  evaluating  the  selected  models  at  the  Fleet  Training 
Center,  Norfolk,  VA.  An  important  recommendation  from  the  Test  and  Evaluation 
conducted  during  Phase  III  was  that  DOTS  should  investigate  model  applications 
at  higher  command  levels. 

Phase  IV  responded  to  the  recommendations  of  Phase  III  by  (1)  operating  and 
testing  at  TRAPAC,  San  Diego,  CA,  the  models  developed  in  Phase  II,  and  (2) 
developing  a new  Training  Requirements  Analysis  Model  (TRAM)  and  field  testing 
it  at  CNTECHTRA,  Memphis,  Tenn. 

The  major  effort  supporting  Phase  II-B  was  the  ETAM  procedural  development 
during  Phase  II-A.  In  addition  to  the  procedural  development,  computer 
applications  were  described,  scenarios  using  the  ETAM  procedures  were  pre- 
pared, and  ETAM  validity  and  logical  structure  were  confirmed. 
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FOREWORD 


The  Design  of  Training  Systems  (DOTS)  project  objectives  are  In  conso- 
nance with  the  requirements  of  Advanced  Development  Objective  ZPN07 
(formerly  ADO  43-03X),  Education  and  Training  Development.  ZPN07  In- 
cludes a number  of  projects  concerned  with  demonstrating  and  evaluating 
the  technical,  operational  and  financial  feasibility  of  applying  advanced 
technological  applications  to  Improving  the  training  process. 

The  Bureau  of  Naval  Personnel  Initiated  the  original  ADO  In  1966  to  make 
naval  training  more  responsive  to  the  changing  times.  As  one  project 
under  this  effort,  DOTS  was  designed  to  improve  the  process  of  managing 
training  resources  through  application  of  the  techniques  of  system  analysis 
and  system  simulation  as  accomplished  through  mathematical  modeling.  The 
end  objective  Is  a family  of  computerized  mathematical  models  enabling 
training  management  to  more  rapidly  predict  the  Impact  of  changes  in  train- 
ing resource  availability  or  requirements. 

The  majority  of  education  and  training  was  reorganized  In  1971  under  one 
command.  Chief  of  Naval  Education  and  Training  (CNET).  Because  of  this 
change,  DOTS  responsibility  was  transferred  to  CNET  In  March  of  1972; 
more  specifically,  to  the  Training  Analysis  and  Evaluation  Group  (TAEG), 
Orlando,  Florida.  The  new  CNET  organization  greatly  Increased  the  potential 
benefits  to  be  gained  from  the  Increased  application  of  new  management  tech- 
niques and,  therefore,  from  the  DOTS'  R&D  effort.  DOTS  began  in  February 
of  1973,  with  the  majority  of  tasking  being  contracted  to  the  International 
Business  Machines  Corporation,  Federal  Systems  Division,  Cape  Kennedy 
Facility,  located  at  Cape  Canaveral,  Florida. 

The  Training  Analysis  and  Evaluation  Group,  Dr.  A.  Smode,  Director,  project 
team  members  Messrs.  M.  Middleton  and  W.  Lindahl,  complemented  the  con- 
tracted effort  by  providing  direction  and  guidance  and  In  establishing 
organizational  Interfaces. 

The  model  was  developed  by  Mr.  L.  R.  Duffy,  Dr.  R.  B.  Miller,  and  Mr.  J.  D. 
Staley.  Mr.  R.  E.  Hallman  and  Hr.  L.  R.  Duffy  provided  management  for  the 
project. 
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SECTION  I 
INTRODUCTION 


BACKGROUND 

The  benefits,  costs  and  r’sks  associated  with  introducing  technical  innova- 
tions in  education  and  training  can  invoke  large  commitments  of  resources. 

The  rational  assessment  of  payoffs  and  penalties  for  investing  in  changes 
therefore  deserves  technical  attention  with  the  objective  of  arriving  at 
sound  decisions  to  accept  or  reject.  In  essence,  this  means  making  an 
analysis  of  the  full  potential  range  of  applicability  of  the  proposed  in- 
novation in  terms  of  benefits,  liabilities  and  risks,  including  financial 
analysis  of  costs,  and  synthesizing  the  mass  of  resulting  data  into  a 
decision-making  presentation.  Key  judgmental  operations  should,  of  course, 
be  retained  by  humans.  Explicitness  in  procedure  and  in  the  expression  of 
human  judgment  is  a key  factor  in  rationality.  The  Educational  Technology 
Assessment  Model  (ETAM)  is  a set  of  comprenensi ve  procedures  and  variables 
for  this  analysis,  synthesis  ana  decision  making.  Although  the  content 
of  this  structure,  or  "model,"  is  directed  specifically  towards  education 
and  training,  the  structure  itself  is  applicable  to  any  rational,  decision- 
making context. 

An  innovation  is  broadly  defined.  In  ETAM  it  is  “a  relatively  constant  or 
enduring  change  in  the  procedures,  objects  or  functions  used  in  any  aspect 
of  the  instructional  process  which  may  be  viewed  as  a benefit  (or  a lia- 
bility) and  has  associated  costs."  Thus,  an  innovation  may  be  a technical 
invention  or  it  may  be  a structural  change  in  the  setting  of  instruction, 
such  as  from  shore-based  to  ship-based  training.  In  summary,  innovations 
range  across  content  of  instruction,  instructional  procedure,  student  selec- 
tion, and  generation  and  implementation  of  training  requirements.* 

The  initial  ETAM  study  developed  a complete  set  of  manual  procedures,  para- 
meters and  formats  for  all  analysis,  synthesis  and  decision.  A comprehensive, 
descriptive  taxonomy  of  educational  technology  was  generated.  Its  purpose  was 
to  enable  any  proposed  innovation  to  be  described  in  a standardized  terminology 
for  determining  the  full  range  of  potential  effect  in  the  Navy:  students, 

courses,  jobs,  instructional  devices,  instructional  development,  as  areas  of 
relevance.  An  equivalent  effort  was  spent  in  developing  or  adapting  analytic 
cost  models  applicable  to  the  Navy's  training  environments. 

The  procedural  model  also  included  return  on  investment  analysis,  sensitivity 
analysis,  and  scaling  procedures  for  translating  subjective  evaluations  of 


* For  additional  reference  to  educational  innovations  see: 

Miller,  Robert  B.  and  Duffy,  Larry  R. , 1975.  Design  of  Training  Systems 
Phase  1 1 - A Final  Report.  TAEG  Report  No.  12-3,  Training  Analysis  and 
Evaluation  Group,  Orlando,  FL.  Chapter  III. 
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relative  worth  into  utility  values.  These  utility  expressions  became  inputs 
to  formal  decision  tree  models  to  be  presented  to  the  executive  decision 
maker.  A logical  flow  of  information  is  maintained  from  one  procedural  step 
to  the  next,  so  that  the  assessor  can  readily  review  the  sources  of  benefits 
data,  cost  data,  judgments  and  assumptions  leading  up  to  the  final  presenta- 
tion of  evaluation  decision  alternatives.  The  decision  maker  can  change  the 
input  values  to  the  sensitive  parameters  in  the  model  and  determine  the  effect 
on  the  decision  alternatives.  The  decision  maker  is  not  restricted  to  work- 
ing only  from  the  formal  outcomes  of  staff  evaluations. 


ASSUMPTIONS  ABOUT  USING  THE  MODEL 

Some  assumptions  that  were  stated  in  the  initial  ETAM  study  should  be 
repeated  here. 

1.  The  primary  user  of  ETAM  will  be  the  assessor  of  the  innovation. 
He  (or  the  assessing  team)  have  expertise  in  the  subject  matter 
of  the  particular  innovation  to  be  evaluated.  He  has  mastered 
the  ETAM  classification  structure,  at  least  to  the  level  of 
being  able  to  reference  its  content.  Furthermore,  he  will  have 
had  at  least  several  dozen  hours  of  preliminary  practice  in 
applying  the  procedures  manually  (except  for  calculational 
problems)  and  can  "walk  his  way  through"  the  major  ETAM  tasks. 

He  will  also  have  background  in  the  operational  aspects  of  Navy 
training  courses,  instructional  devices  and  media.  Navy  jobs, 
and/or  the  developmental  stages  of  training;  at  least  he  will 
be  familiar  with  those  aspects  of  these  operations  relevant 
to  the  innovation.  This  assumption  recognizes  that  humans 
will  provide  the  information  inputs  and  judgments,  whereas 
the  model  merely  structures,  guides,  and  within  defined  limits 
processes  them. 

2.  The  secondary  user  of  ETAM  will  be  the  executive  decision  maker. 
He  makes  the  decision  to  commit,  deny  or  commute  the  resource 
for  implementing  the  decision.  He  may  question  the  constituent 
or  summary  judgments,  evaluations,  data  sources  and  predictions 
of  the  assessor  embodied  in  the  final  evaluative  recommendation. 
He  can  "peel  back"  the  various  layers  of  data  and  judgements 
entering  into  the  final  calculations.  ETAM  documentation  should 
facilitate  this  normal  relationship  between  the  executive  and 
his  advisory  staff  work.  The  service  of  the  computer  should 
aid  rather  than  hinder  this  inspection. 

A key  factor  is  the  ability  to  identify  the  factors  most  sensi- 
tive to  the  decision  outcome,  and  test  the  range  of  this  sensi- 
tivity across  the  limit  where  a recommended  decision  choice  A 
changes  to  a decision  choice  B. 

3.  The  assessor  will  make  the  final  judgments  as  to  whether  the 
innovation  is  or  is  not  applicable  to  an  entity  such  as  a 
given  training  course,  a given  instructional  vehicle,  a given 
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job  or  job-task.  The  range-of-effect  search  operations,  in- 
cluding the  indexing  of  the  innovation  and  indexing  the  data 
base  content,  and  automatic  searches  on  data  bases,  will  facili- 
tate those  judgments,  but  not  replace  them. 

4.  The  user  is  not  compelled  to  apply  the  procedures  beyond  any  stage 
where  common  sense  shows  that  the  outcome  will  be  hopelessly  nega- 
tive. The  model  is  segmented  into  stages  so  as  to  help  this  kind 
of  efficient  termination  come  about.  It  is  easy  to  become  disen- 
gaged from  the  model  with  a sensibly  completed  piece  of  work.  The 
assessor  is  also  encouraged  to  examine  the  outcome  of  each  stage  of 
work  for  being  within  the  bounds  of  reasonableness. 

OBJECTIVES  FOR  ETAM 

The  design  phase  of  ETAM  led  to  the  specification  of  a set  of  manual  procedures 
for  assessing  a proposed  innovation  or  change  in  Navy  training.  A proper 
follow-on  objective  was  adapting  computerized  aids  for  reducing  the  large 
burdens  of  manual  activities  and  supplementing  rather  than  replacing  or  inter- 
fering with  human  judgmental  processes.  The  scaling  operations  leading  to 
expressions  of  utility  in  decision  models  were  so  important  that  they  deserved 
intensive  study  of  theory  and  practice  in  behavioral  utility  models  before 
adopting  and  standardizing  on  any  given  procedure.  Another  key  issue  was 
the  practicability  of  the  ETAM  classification  structure  to  the  indexing  and 
searching  of  the  content  in  Navy  data  bases  dealing  with  training  courses, 
jobs  and  job-tasks,  instructional  vehicles  and  media.  These  issues  were  the 
basis  for  the  next  phase  of  ETAM. 

The  following  tasks  summarize  the  initial  ETAM  objectives: 

1.  A study  to  determine  appropriate  scaling  techniques  which  would 
increase  the  expected  reliability  and  validity  of  subjective 
estimates  required  within  the  ETAM  procedures.* 

2.  A study  to  define  indexing  methods  to  provide  equivalence 
between  the  ETAM  range-of-effect  taxonomic  elements  and  data 
base  descriptors  for  the  purpose  of  achieving  effective  data 
search  and  retrieval  operations.** 

3.  A major  report  presenting  a comprehensive  overview  of  the  innova- 
tions in  concepts,  methods  and  practices  that  have  shaped  and 
are  currently  influencing  modern  instructional  technology.  The 
purpose  was  to  give  the  intelligent  layman  an  overview  of  the 
effects  of  innovation  upon  training.  Such  a document  would 

aid  high-ranking  military  officers  and  business  executives 


* Miller,  Robert  B.,  and  Duffy,  Larry  R.,  1975.  Design  of  Training  Systems, 
The  Development  of  Scaling  Techniques.  TAEG  Report  No.  32,  Training 
Analysis  and  Evaluation  Group,  Orlando,  FL. 

**  Refer  to  Appendix  A of  this  report,  TAEG  Report  No.  40. 
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in  making  decisions  on  applying  proposed  innovative  techniques 
and/or  technologies  to  their  training  system.  There  are  parallels, 
but  few  direct  equivalents,  of  the  content  of  the  initial  ETAM 
report  (TAEG  Report  No.  12-3).  It  would  be  an  appropriate  orient- 
ing background  to  the  ETAM  report  for  those  using  its  content.* 

4.  Computerization  of  the  ETAM  logic  so  that  an  assessor  can  inter- 
actively arrive  at  an  accept/reject  conclusion  using  various 
utility,  probability,  cost  and  benefit  data  as  model  inputs. 
Standard  program  documentation  and  a user's  guide  will  be  pro- 
duced as  an  output  of  this  task. 


* Miller,  Robert  B. , and  Smode,  Alfred  F.,  1976.  Major  Innovations  in 
Training  Technology.  Training  Analysis  and  Evaluation  Group,  Orlando, 
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SECTION  II 

ETAM  FUNCTIONAL  REQUIREMENTS 


This  section  of  the  report  has  two  components.  The  first  deals  with  the  pro- 
cedures arising  from  the  steps  in  the  operational  sequence  of  ETAM.  Each  of 
these  steps  has  a set  of  defined  information  inputs,  processing  activities,  and 
defined  information  outputs.  This  procedural  definition  was  presented  in  the 
initial  ETAM  report,  but  subject  to  modification  by  the  results  of  the  studies 
on  scaling  and  on  indexing  procedures.  The  second  component  of  the  functional 
requirement  is  the  design  requirement  for  computer  support  to  the  procedures. 

PROCEDURAL  REQUIREMENTS 

The  ETAM  procedures  consist  of  eight  major  tasks.  The  following  descriptions 
will  focus  on  the  data  management  components  of  these  tasks  rather  than  on  how 
they  are  performed  since  these  aspects  are  most  relevant  to  the  functions  of 
computerized  data  storage,  processing  and  retrieval. 

Figure  II-l  is  a schematic  of  these  steps. 

TASK  1 - FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION.  A project  file  is  ini- 
tiated to  serve  as  a data  base  for  Innovation  X.  The  innovator  identifies  the 
objectives  of  the  innovation  as  he  conceived  them,  target  applications,  and  the 
results  of  empirical  studies,  if  any,  made  from  the  innovation,  or  cited  as 
relevant  to  the  innovation.  With  the  assistance  of  a staff  "assessor"  with  ETAM 
background,  the  prose  description  of  the  innovation  is  indexed  according  to  the 
taxonomic  classification  descriptors  in  ETAM.  The  indexed  innovation  as  a set 
of  descriptors  will  be  used  as  a search  specification  against  data  bases  in  Task 
5--determination  of  range-of-effect.  The  information  in  prose  form  will  be 
retained  in  a Task  1 file. 

TASK  2 - DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION.  In  this  task  the 
innovator  (or  other  expert)  is  requested  to  consider  possible  alternatives  to 
the  proposed  innovation  which  may  require  a lesser  level  of  investment  funding, 
and  possibly  be  more  cost  effective.  Any  outcome  of  Task  2 will  be  treated 
procedurally  like  Task  1 and  the  outcomes  of  Task  1.  The  ultimate  result  will 
be  to  create  a decision  that  compares  Innovation  X with  the  alternative  Innova- 
tion XX.  The  profile  of  descriptors  that  indexes  Innovation  XX  may,  or  may  not, 
be  identical  to  the  index  of  Innovation  X.  Innovation  XX  will  generate  its  own 
file  with  a direct  associative  link  to  Innovation  X. 

TASK  3 - MAKE  PRELIMINARY  FEASIBILITY  PROFILE.  The  formal  assessment  procedure 
begins  at  this  point  with  a questionnaire  about  potential  risks  in  the  imple- 
mentation, acceptance  and  application  of  the  innovation.  The  issues  deal  with 
organizational  incompatibilities,  goal/policies  incompatabilities,  technical 
support  requirements,  funding  constraints  and  problems  in  attitudinal  acceptance 
by  users.  A format  is  available  for  entering  risk  estimates  and  comments  on 
each  of  the  key  risk  variables. 


II-l 


TAEG  REPORT  NO.  40 


FIGURE  11-1.  ETAM  PROCEDURAL  SEQUENCE 
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If  the  risks  are  high,  risk  reduction  projects  are  formulated.  Their  costs  are 
roughly  estimated  and  the  consequent  risk  reduction  is  also  estimated.  If  the 
overall  risks  still  seem  excessively  high,  the  decision  to  reject  the  innovation 
from  further  consideration  may  be  made  at  this  point. 

A format  for  collecting  these  data  is  contained  in  Appendix  B of  this  report 
(Figure  B-l).  The  content  becomes  part  of  the  Project  File.  It  will  be  used  in 
Task  4 and  appear  as  essential  data  in  Task  8. 

TASK  4 - PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT.  The  results  of  Task  3 are 
analyzed  in  greater  depth  and  risk  reduction  studies  and  projects  are  further 
defined  and  costed.  A preliminary  decision  tree  is  structured  for  providing 
initial  guidance  as  to  whether  the  innovation  should  be  accepted  outright, 
accepted  with  the  additional  expenditures  for  the  risk  reduction  projects,  or 
rejected.  Presumably,  a definitive  range-of-effect  study  of  potential  benefits 
nas  not  yet  been  justified,  or  a sample  of  already  known  target  applications  for 
the  innovation  is  a sufficient  working  basis  for  this  stage  of  assessment. 

Project  descriptions  are  prepared  for  each  R&D  effort  with  supporting  data  about 
resource  requirements,  cost  analysis,  and  time  schedules.  These  projects  are 
grouped  into  packages,  each  of  which  is  intended  to  reduce  the  overall  risk  to  a 
reasonable  level.  A format  for  collecting  cost/saving  data  is  shown  in  Appendix 
B (Figure  B-2) . 

Decision  trees  are  developed  from  estimated  benefits  data,  cost  data  associated 
with  various  supplemental  projects,  and  risk  estimations.  Refer  to  Appendix  B 
(Figures  B-3  and  B-4) . 

Note  that  at  this  stage,  the  range-of-effect  and  cost  analysis  has  been  only 
grossly  estimated  rather  than  derived  from  a full  scale  analysis.  But  even  on 
these  bases,  the  differences  among  the  decision  alternatives  may  be  so  large 
and,  based  on  sensitivity  analysis,  appear  so  reliable,  that  a decision  may  be 
justified  without  further  analysis. 

If  analysis  proceeds  further,  the  risk  estimations  and  risk  reduction  project 
data  are  fed  into  Task  6. 

TASK  5 - DETERMINE  RANGE-OF-EFFECT.  The  prior  tasks  have  been  concerned  with 
the  assessment  of  the  innovation  over  a limited  range  of  application.  Ini- 
tially, it  was  the  target  applications  identified  by  the  innovator;  in  Task  4 a 
preliminary  extended  range-of-appl ication  was  considered.  Task  5 enables  the 
assessor  to  apply  both  the  formal  descriptors  applied  to  indexing  the  innovation 
and  the  contextual  knowledge  about  the  innovation  to  the  full  range  of  the 
Navy's  inventory  of  training  courses,  instruction  vehicles  and  job-tacks. 

ETAM  stipulates  that  entities--training  courses,  job-tasks,  instructional 
vehicles--subject  to  range-of-effect  analysis  may  be  indexed  according  to  the 
rules  of  the  descriptor  taxonomy  in  ETAM  formulated  in  Appendix  A of  this 
report.  These  indexed  entities  comprise  a data  base.  This  data  base  can  be 
searched  by  search  arguments  composed  of  the  descriptors  that  uniquely  identify 
the  relevant  properties  of  the  innovation. 
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Thus,  the  indexed  description  of  the  innovation  that  was  made  in  Task  1 becomes 
an  input  into  range-of-effect  search.  When  the  search  arguments  reveal  hits  in 
the  data  base,  the  assessor  examines  contextual  information  about  the  entity  (a 
course,  a job-task,  or  an  instructional  vehicle)  and  makes  a judgment  of  rele- 
vance or  irrelevance.  When  matches  between  the  innovation's  properties  and  the 
entity's  properties  are  made,  the  assessor  estimates  the  kind  and  proportional 
magnitude  of  benefit/liability  that  is  likely  to  be  contributed  by  the  innovation. 
A method  from  the  initial  ETAM  study  for  describing  a benefit  (or  liability)  is 
shown  in  Appendix  B (Figure  B-5).  Ordinarily,  there  will  be  a number  of  benefit 
variables  and  liability  variables  that  make  up  a pattern  or  profile  applicable 
to  the  innovation's  promise.  The  identification  of  affected  entities  (courses, 
etc.)  is  input  to  cost-benefit  analysis,  Task  6. 

TASK  6 - PERFORM  COST-BENEFITS  ANALYSIS.  The  decision  tree  developed  in  Task  4 
is  refined  to  include  more  precise  costs  and  savings  derived  from  processing  the 
tangible  benefits  through  the  model.  Thus,  if  the  innovation  has  been  estimated 
to  enable  an  average  reduction  of  20%  for  learning  the  content  of  Course  A to 
criterion,  the  cost  model  determines  how  frequently  the  course  is  taught,  how 
many  students  take  the  course  and,  from  its  base  of  cost  data  about  Course  A, 
computes  in  dollars  the  actual  projected  savings.  In  addition,  the  assessor 
uses  utility  scaling  techniques  for  analyzing  intangible  benefits  so  that  they 
are  expressed  in  "equivalent  dollars,"  thus  enabling  them  to  be  combined  into  a 
single  continuum  of  worth  or  value.  Equivalent  dollars  is  a utility  expression 
rather  than  a literal  dollar  value.  Probabilities  of  implementation  success  and 
user  acceptance  with  and  without  the  risk  reduction  projects  are  refined  and  the 
decision  variables  are  recalculated. 

The  procedural  model  described  in  the  ETAM  Phase  II-B  report,  "Design  of  Train- 
ing Systems,  The  Development  of  Scaling  Procedures,"  is  the  structure  whereby 
the  assessor  generates  multivariate  utilities  for  outcomes  in  the  decision  tree. 

The  model  permits  sensitivity  analysis  of  variables  that  could  reasonably  change 
enough  to  affect  the  choice  of  a decision  alternative. 

The  output  of  this  Task  goes  into  Task  7,  and  subsequently  to  the  decision  maker 
in  Task  8. 

TASK  7 - PERFORM  FINANCIAL  ANALYSIS.  This  task  is  concerned  with  assessing  the 
tangible  benefits  and  liabilities  (those  expressable  in  real  dollars)  in  terms 
of  certain  economic  measures.  The  investment  costs  and  the  annual  costs  and 
savings  are  calculated  over  a planning  period  extending  a number  of  years  into 
the  future.  Rates-of-return  on  the  invested  dollars  are  determined  for  the 
incremental  effect  of  each  alternative  compared  to  the  primary  project.  Alter- 
natives consist  of  the  proposed  innovation,  the  existing  system,  and  any  other 
approaches  defined  in  Task  2 which  were  considered  reasonable  candidates  Tor 
further  assessment.  This  assessment  process  provides  a separate,  distinct  view 
of  the  value  of  the  innovation  from  that  gained  from  the  decision  tree  assess- 
ment in  Task  6.  Both  are  inputs  to  the  decision  maker. 

The  model  enables  sensitivity  analysis.  The  purpose  is  to  give  the  assessor 
insight  into  the  variables  that  could  cause  a change  in  the  decision  if  they 
were  to  vary  over  a reasonably  expected  range. 
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TASK  6 - MAKE  THE  ACCEPT/REJECT/STUDY  DECISION.  The  immediate  bases  for  the 
making  of  the  decision  are  the  financial  analysis,  plus  sensitivity  analysis, 
from  Task  7,  and  the  decision  tree  data,  plus  sensitivity  analysis,  from  Task  6. 

However,  the  organized  content  of  the  data  base  files  of  the  assessment  project 
enables  the  decision  maker  to  examine  any  of  the  constituent  elements  beneath  the 
summary  presentation  made  to  him.  He  may  substitute  his  own  evaluations  of 
worth,  probability  of  outcomes,  importance  of  intangibles,  estimates  of  benefits 
or  liabilities.  He  may  "peel  back"  the  data  in  each  of  the  seven  tasks  by 
selectively  accessing  the  files  on  each  of  these  tasks.  He  could  examine  samples 
of  range-of-effect  entities  contained  in  computer  files  and  retrieved  interactively. 

The  executive  is  thus  in  a position  to  put  probes  behind  the  facade  of  con- 
clusions presented  to  him.  He  is  therefore  capable  of  reassurance  in  the  results 
or  direct  participation  in  changing  them  according  to  his  own  values  and  store 
of  information. 

COMMENT.  The  preceding  description  is  merely  a synoptic  outline  of  the  ETAM 
procedure.  It  is  neither  a substitute  nor  replacement  for  the  full  description, 
including  stipulations,  assumptions  and  caveats,  that  are  contained  in  the 
source,  the  initial  ETAM  report,  TAEG  Report  No.  12-3. 

DESIGN  REQUIREMENTS 

The  Education  Technology  Assessment  Model  as  outlined  in  the  previous  section  is 
an  organized  grouping  of  procedural  sequences  that  provide  a structure  cor 
efficiently  and  effectively  assessing  the  true  value  of  a proposed  innovation  in 
training.  Efficiency  is,  however,  highly  dependent  upon  the  assessor's  ability 
to  access  data  when  required,  rapidly  make  calculations  using  selectable  routines 
(e.g..  Training  Cost  Model),  modify  input  data  and  make  recalculations,  and 
obtain  output  reports  with  suitable  content  and  format  for  effective  analysis 
ana  decision  making.  An  efficient  assessment  processing  structure  requires: 

1.  Organized  data  bases  external  to  ETAM  with  rapid  search  and 
retrieval  capability. 

2.  Automated  routines  which  can  be  selected  at  specific  stages 
of  the  assessment  process  for  making  calculations  or  manipu- 
lating data. 

3-  Project  files  which  will  serve  as  a repository  for: 

Descriptive  data  on  the  innovation. 

Input  data  factors. 

Results  of  data  base  searches. 

Intermediate  calculated  results. 

Implementation  subproject  descriptive  and  cost  data. 

Final  analytic  results. 

4.  An  organized  system  of  manual  and  computerized  steps  that  will 
make  it  possible  for  the  trained  user  to  select  prudently  the 
sequencing  of  data  retrieval,  data  manipulation,  calculations, 
interactive  requirements,  and  output  generation  based  upon  the 
analytic  needs  at  a particular  phase  of  the  assessment. 
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OVERALL  SYSTEM  OBJECTIVES.  The  user  is  a major  concern  in  the  development  of 
any  complex  interactive  automated  data  system.  The  following  objectives  are 
intended  as  overall  guidance  to  the  development  process  to  ensure  a high  level 
of  user  orientation  is  achieved. 

1.  Operational  Ease. 

There  will  be  minimal  requirement  that  the  ultimate  user  be 
familiar  with  both  the  software  and  hardware  design  in  order  to 
solve  his  problems  using  the  system.  This  implies  an  interface 
with  the  computer  in  a language  which  is  presently  within  the 
user's  repertory,  and  sufficient  prompting  and  assistance  to 
promote  simplicity  in  problem  entry  as  well  as  in  output  inter- 
pretation. 

2.  Interactive  Capability. 

Much  of  the  data  entry  initially  will  be  in  batch  form;  however, 
once  the  basic  analytic  process  has  begun,  a high  degree  of  user 
interaction  will  be  a part  of  the  overall  design  structure.  The 
user  will  be  able  to  specify  solution  formats,  to  modify  and  over- 
ride existing  data,  to  call  upon  specific  calculation  or  data 
manipulation  routines,  or  to  specify  output  report  formats.  The 
assistance  provided  the  user  through  on-line  prompting  will  be 
sufficient  to  permit  rapid  convergence  toward  problem  solution 
and  decision. 

3.  Modular  Design. 

As  the  user  undertakes  the  assessment  process,  additional  calcu- 
lation and  data  manipulation  .requirements  may  be  identified.  The 
software  system  will  be  designed  in  such  a way  that  existing  rou- 
tines can  be  modified  or  deleted  with  minimal  impact  on  the  other 
components  of  the  system  and  so  that  additional  routines  can  be 
developed  and  incorporated  as  a new  module  within  the  existing 
system  framework.  This  facility  is  necessary  to  maintain  a 
"general  purpose"  character  to  the  assessment  process,  so  that 
analytic  processes  unique  to  assessing  certain  types  of  innova 
tions  can  be  added  as  they  are  identified. 

4.  Data  Base  Maintenance. 

Several  data  bases  have  been  identified  to  interface  with  the 
ETAM  programs.  The  intent  is  to  access  existing  Navy  data  bases, 
insofar  as  is  possible,  so  that  existing  data  base  maintenance 
approaches  can  be  used.  The  objective  is  to  minimize  the  amount 
of  additional  data  to  be  collected  and  stored. 

ETAM  AUTOMATED  DATA  SYSTEM  FUNCTIONAL  DESIGN.  The  automated  data  system  is 
designed  with  five  major  functions  to  control  the  flow  of  data  and  sequence  of 
calculations,  as  well  as  provide  input  data  paths  to  the  data  bases.  The  inter- 
relationships between  these  five  functions  and  the  data  bases  are  shown  in 
Figure  1 1-2 . Following  are  descriptions  of  each  function  and  the  types  of 
programs  or  subfunctions  which  may  be  contained  within  them.  Specific  programs 
and  file  structures  developed  to  fulfill  these  functions  are  detailed  in  Section 
III  of  this  report. 
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Description  Structuring.  Source  descriptive  and  project  data  are  entered  via 
the  Description  Structuring  program  Project  identifications  are  established 
to  ensure  traceability  throughout  the  entire  analytic  and  data  transfer  process. 
Overall  data  structuring,  as  it  relates  to  the  organization  of  the  project  file 
data  base,  is  performed  here.  Any  batch  data  input  (e.g.,  cost  model  input 
parameters)  is  entered  through  one  of  the  Description  Structuring  programs. 

Objective:  The  Description  Structuring  programs  (DSP)  establish  a new  project 

region  within  the  project  file  section  of  the  ETAM  Working  Data  Base.  They 
accept  the  total  range  of  descriptive  data  on  the  new  project  (proposed  training 
innovation)  and  format  it  for  subsequent  processing.  They  also  accept  major 
data  subsets  associated  with  the  project  which  will  include:  (1)  experimental 

studies  with  descriptions  and  resultant  data,  (2)  risk  reduction  implementation 
projects  with  descriptions,  resource  requirements,  schedules,  and  impacts  upon 
overall  project  succcess,  (3)  alternative  project  descriptions,  and  (4)  unique 
costing  data.  Input  to  the  DSP  will  define  the  initial  processing  sequence  and 
output  requirements. 

Input:  Project  file  initialization  and  most  updating  will  be  performed  via  card 

input.  The  major  input  sets  will  be: 

1.  Project  descriptive  data. 

2.  Benefit  pattern  data. 

3.  Risk  profiles  and  risk  reduction  project  data. 

4.  Decision  tree  parameters  (outcome  scenarios,  variables,  etc.). 

5.  Alternative  projects  descriptive  data. 

Processing  Functions:  The  DSP  will  accept  input  data  sets.  The  required  file 

management  routines  will  be  called  and  assigned  to  process  the  input  which  may 
be  new  data  or  replacement  data.  Once  data  sets  have  been  assigned,  a project 
table  will  be  created  which  identifies  all  data  sets  related  to  the  primary 
project  seing  proposed. 

Output:  The  major  DSP  output  are  the  individual  files  created  within  the  pro- 

jected data  base. 

Range-of-Effect.  The  purpose  of  these  programs  is  to  formulate  the  processing 
pattern  which  is  to  form  the  input  for  the  Assessment  Programs.  The  user  is  to 
be  provided  preliminary  guidance  in  preparing  the  taxonomic  descriptors  which 
will  be  used  to  search  out  the  innovation's  full  range  of  application.  Once  the 
search  characteristics  have  been  defined,  either  interactively  or  off-line,  the 
specific  search  paramater  for  the  equipment,  course,  or  job  data  bases  are 
generated.  The  ETAM  abbreviated  data  base  which  contains  abbreviated  records 
for  Navy  courses,  jobs,  etc.,  and  user  interactive  inputs  are  the  primary  source 
of  data  upon  which  the  Range-of-Effect  programs  operate.  Any  of  the  entities 
extracted  through  a data  search  operation  will  be  passed  to  the  extracted  data 
base  for  further  manipulation. 
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Objective:  Primary  objectives  of  the  Range-of-Effect  programs  (REP)  are  (1)  to 

create  the  necessary  set  of  data  base  search  parameters  for  obtaining  the  enti- 
ties (e.g.,  course,  jobs)  to  which  the  innovation  has  potential  application,  and 
(2)  to  structure  the  benefit  pattern  used  by  the  Assessment  function  in  calcu- 
lating the  decision  variables. 

Input:  Project  descriptive  data  from  tne  project  file  will  provide  the  target 

applications  for  the  proposed  innovation.  This  will  establish  the  initial 
descriptor  content.  Benefit  estimates  and  the  empirical  data  will  define  bene- 
fit patterns  and  allocation  rules  for  further  modifying  the  search  parameters. 

Processing  Functions:  The  appropriate  abbreviated  data  base  may  be  searched 

based  upon  the  target  application  and  the  descriptor  index  will  be  presented  for 
the  review  of  the  assessor.  A conversion  table  will  translate  numeric  indices 
to  appropriate  elements  of  the  taxonomies  related  to  the  entity  class  for  the 
target  application.  User  Interactive  programs  (UIP)  will  prompt  the  assessor 
and  select  taxonomic  menus  to  guide  this  refinement  process.  Multiple  indices 
may  be  developed  if  improvements  of  the  taxonomic  menus  are  OR'ed  together. 

Once  the  descriptor  indices  have  been  formulated,  the  appropriate  abbreviated 
data  base  will  be  searched  to  obtain  all  entities  which  match. 

Allocation  rules  are  for  the  purpose  of  further  refining  the  list  of  selected 
entities.  For  example,  if  courses  having  a length  of  one  week  or  more  can 
benefit  from  the  proposed  innovation,  this  would  become  an  additional  selector 
factor  when  the  list  of  entities  is  forwarded  to  the  Navy  data  base  controlling 
function  for  extraction  of  the  full  range  of  attributes  associated  with  the 
entities.  The  assessor  must  determine  if  the  additional  selectors  are  to  be 
used  to  limit  the  number  of  entities  extracted  or  whether  they  should  be  applied 
to  the  total  set  of  extracted  entities  once  they  have  been  loaded  in  the  extracted 
data  section  of  the  ETAM  Working  Data  Base.  This  aetermi nation  would  be  based 
upon  whether  the  entities  eliminated  by  the  selector  would  ever  be  brought  into 
the  Assessment  program  analysis  from  a change  in  allocation  rules. 

A final  processing  function  of  the  REP  is  the  formulation  of  the  benefit  pattern 
used  to  input  various  routines  with  the  Assessment  Program.  The  ETAM  procedures 
for  Tasks  4,  5 and  6 should  provide  sufficient  guidance  on  the  content  and 
format.  Some  benefit  patterns  will  involve  directly  costable  functions  (i.e., 
straight  displaceable  costs)  such  as  in  the  case  where  equipment  is  eliminated 
or  replaced. 

Output:  The  Range-of-Effect  Program  develops  an  output  report  containing  a 

list  of  entities  and  selector  factors  to  be  used  in  searching  existing  Navy  data 
bases  to  obtain  the  full  attribute  set  for  each  entity  matched.  The  second 
output  will  be  a table  of  variables,  degree  of  change,  importance,  confidence, 
etc.,  which  will  represent  the  benefits  pattern  to  be  used  within  tht  Assessment 
Program.  Allocation  rules  which  were  not  used  in  generating  selector  factors  on 
the  entity  extraction  tape  will  also  be  stored  for  use  by  the  Assessment  Program. 

Assessment.  The  Assessment  programs  perform  computational  steps  in  the  analytic 
process.  This  encompasses  four  basic  types  of  computational  capabilities; 
namely,  cost  models,  decision  tree  evaluators,  financial  analysis  routines,  and 
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sensitivity  analysis  routines.  In  conjunction  with  these  computational  routines, 
basic  input  editing  and  default  data  selection  functions  are  carried  out  by  the 
Assessment  programs.  These  routines  will  communicate  with  the  user  to  identify 
additional  data  requirements,  or  to  point  out  the  characteristics  of  particular 
default  data  being  used.  Final  results  will  be  retained  in  the  ETAM  project 
file  for  additional  processing  or  modification  if  desired.  An  aggregation 
routine  will  provide  the  means  of  summarizing  the  data  from  multiple  selected 
entities  (e.g.,  jobs),  so  that  the  computational  routines  can  be  run  on  aggre- 
gated data. 

Objective:  The  Assessment  programs  (AP)  provide  the  data  manipulation  and 

computational  capability  for  generating  costs,  benefit  quantities,  decision 
variables,  and  sensitivity  factors.  These  become  the  key  inputs  to  the  decision 
maker.  The  AP  will  have  the  capability  of  aggregating  and  weighting  entities 
either  for  inputting  cost  models  or  resulting  from  processing  by  a cost  model. 
Decision  variables  calculated  on  aggregated  data  will  be  obtained  from  either  a 
cost-benefits  analysis  using  a decision  tree  framework  or  from  a financial 
return-on-investment  analysis. 

Input:  Control  instructions  from  the  User  Interactive  programs  (UIP)  will  guide 

the  processing  sequence  for  the  AP;  thus,  the  input  factors  selected  will  be  a 
function  of  these  instructions.  Each  routine  (training  costs  model,  job  cost 
model,  decision  tree  generator,  etc.)  will  require  a set  of  data  variables  which 
will  be  stored  in  the  extracted  data  section  of  the  ETAM  project  files.  Each 
routine  will  also  have  provision  for  input  editing  and  for  the  generation  or 
selection  of  default  data  in  the  cases  where  information  is  missing. 

Processing  Functions:  The  processing  design  was  guided  by  the  initial  ETAM 

study  procedures  for  cost-benefits  analysis  and  financial  analysis.  The  pro- 
cessing routines  within  the  AP  are: 

1 . Training  Cost  Model 

The  basic  logic  of  a Navy  model  developed  for  costing  training  require- 
ments as  a part  of  TECEP  (Training  Effectiveness/Cost  Effectivess  Pre- 
diction Technique)*  was  incorporated  into  the  Training  Cost  Model  speci- 
fied for  ETAM.  Several  additional  features  to  be  incorporated  into  the 
final  developed  program  outlined  in  Section  III  of  this  report  are: 

Extensive  input  data  editing  and  default  data  specification. 

User  selectable  policies  for  equipment  purchase  and  equipment  depre- 
ciation. 

Selectable  output  reports. 

Output  format  design  compatible  with  CRT  display  to  facilitate  any 
conversion  from  the  present  batch  to  an  interactive  system  with  display 
capabi 1 ity. 


*Braby , Richard  (Ed.D),  Henry,  James  M.,  Parrish,  William  F.  Jr.,  and  Swope,  William 
M.  (Ph.D),  1975.  A Technique  for  Choosing  Cost-Effective  Instructional  Delivery 
Systems.  TAEG  Report  No.  16,  Training  Analysis  and  Evaluation  Group,  Orlando,  FL. 
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2.  Job  Cost  Model 

The  costing  of  job  related  impacts  requires  a manual  analysis  of  the 
job/tasks  affected  by  the  innovation.  Inputs  can  be  in  the  form  of 
frequency  of  tasks  and  time  per  task  which  can  be  converted  to  equi- 
valent numbers  of  persons  performing  the  tasks,  or  total  persons  im- 
pacted can  be  input  directly.  Other  costs  which  can  be  estimated  and 
inputted  are: 

Error  rates  and  cost  per  error. 

Support  costs;  e.g.,  tools,  supervision,  etc. 

3.  Equipment  Cost  Model 

The  "instructional  vehicle"  route  in  the  Range-of-Effect  Task  5 is 
taken  when  the  innovation  is  concerned  with  a straight  cost  displacement 
effect.  The  data  base  accessed  for  obtaining  attributes  of  equipments 
will  contain  cost.  The  model  for  determining  equipment  cost  requires 
manual  estimation  of  such  items  as: 

Numbers  affected 
Equipment  purchase  cost 
Purchase  policy 
Equipment  maintenance  cost 
Life  of  equipment 
Depreciation  policy 

4.  Decision  Tree  Evaluator 

The  decision  tree  evaluation  routine  performs  the  necessary  calculations 
required  to  fold  back  the  tree  and  determine  the  value  of  the  decision 
variable. 

The  inputs  to  the  decision  tree  evaluation  will  be  able  to  be  entered 
directly  in  the  batch  mode  or  interactively  through  User  Interaction 
programs.  Inputs  to  this  routine  are: 

Outcome  utilities  (equivalent  costs) 

Success  probabilities 
Risk  package  costs 

5.  Financial  Analysis  Routine 

This  routine  performs  a comparative  return  on  investment  analys's  on  the 
cost/savings  effects  of  implementing  the  proposed  innovation.  Comparisons 
can  be  made  between  the  primary  proposal  and  the  existing  system,  between 
the  primary  proposal  and  any  identified  alternative  proposals,  and  between 
any  two  alternative  proposals.  The  present  value  of  each  comparative 
investment  is  calculated  based  on  an  inputted  discount  rate.  A Savings 
Investment  Ratio,  Rate  of  Return  on  Investment,  and  Uniform  Annual  Cost/ 
Savings  are  calculated  for  each  comparison. 
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6.  Sensitivity  Analysis  Routine 

Two  sensitivity  analysis  routines  are  required.  One  is  used  with  the 
decision  tree  where  the  decision  variable  is  evaluated  for  its  sensitivity 
to  each  variable  included  in  the  tree  outcomes  (A  through  D) , to  the 
success  probabilities,  and  to  the  risk  reduction  package(s)  cost.  As  is 
outlined  in  the  initial  ETAM  procedures,  the  results  of  the  sensitivity 
analysis  will  be: 

Break-even  value 
Decision  change  (from/to) 

Sensitivity  factor 
Probability  factor 

The  mean  value  and  confidence  range  for  each  variable  are  the  inputs  to 
the  sensitivity  analysis  routine. 

A second  sensitivity  analysis  routine  (actually  to  be  developed  as  a 
subroutine  of  the  first)  will  treat  rate-of-return  and  present  value 
decision  variables  as  the  dependent  variables,  and  will  allow  a sensi- 
tivity calculation  to  be  performed  using  each  benefit  variable  and 
discount-rate  as  independent  variables. 

7.  Aggregation  Routine 

The  assessment  program  has  the  facility  for  summing  the  results  of 
evaluating  multiple  entities.  For  example,  if  the  innovation  impacts 
fifty  courses,  each  course  would  be  processed  through  the  training  cost 
model,  and  the  results  aggregated  through  this  routine.  Similar  aggre- 
gations would  be  performed  for  jobs,  equipments,  etc. 

Output:  The  AP  routines  generate  output  data  sets  which  can  be  accessed  by  report 

selection  and  output  structuring  routines.  The  specific  report  formats  are  outlined 
in  Section  III  of  this  report. 

User  Interactive.  Many  of  the  ETAM  software  elements  will  require  initiation  via 
user  on-line  terminal  control.  The  programs  are  "interactive"  in  the  strict  sense 
that  user  input  is  required  to  determine  course  of  action,  operational  modes,  etc. 
The  user  input  will  be  prompted  when  required.  Some  programs  are  capable  of  pro- 
ducing a variety  of  output  reports.  The  selection  of  report  format  and,  in  some 
instances,  the  amount  of  data  to  be  included  in  the  report  will  be  provided  by  the 
user  via  terminal  input.  In  certain  routines,  such  as  the  decision  tree  evaluation, 
interaction  with  the  user  may  be  highly  desirable  in  formulating  parameters  or 
in  generating  the  scenarios  required  for  each  decision  tree  outcome. 

Objective:  Interactive  control  allows  the  assessor  to  maintain  control  over  pro- 

cessing functions  and  input  and  output  both  batch  and  interactive.  Such  control 
extends  to  program  control  and  maintenance  of  the  ETAM  data  bases.  To  be  imple- 
mented will  be  interaction  with  the  assessor  to  perform  editing,  general  prompting, 
data  retrieval,  and  output  formatting. 

Inputs:  Inputs  to  those  processing  functions  which  would  normally  require  user- 

supplied  input  will  be  in  many  instances  supplied  by  two  sources.  Within  the 
executive  control  language  of  the  system  which  calls  the  processing  functions. 


11-12 


TAEG  REPORT  NO.  40 


default  execution  parameters  can  be  supplied;  these  defaults  can  be  overridden 
by  terminal  input.  In  some  instances,  final  input  to  a process  may  involve  a 
combination  of  terminal  input  commands  and  executive  control  defaults. 

Processing  Functions:  User  interaction  will  be  required  for  the  control  of  those 

functions  which  encompass  the  following  ETAM  activities: 

1.  Data  input  and  data  editing.  Specifically,  this  function  is  per- 
formed for  any  processes  which  require  data  to  be  input  from  the 
terminal . 

2.  Prompting:  A number  of  prompting  routines,  including  assistance 

("help")  to  the  user,  guides  for  the  assessor  in  data  input  and 
retrieval,  processing  control  and  decisions,  and  output  selection. 

3.  Output  reporting:  Many  routines  can  supply  a variety  of  distinct 

output  reports.  The  user  may  command  the  type  and,  in  some  instances, 
the  amount  of  on-line  reporting  to  meet  his  immediate  requirements. 

In  some  processes,  both  on-line  (terminal)  and  off-line  (batch)  output 
may  be  selected. 

Output:  The  major  outputs  are  data  exception  reports  from  the  data  editing  functions, 

prompting  instructions  to  the  assessor  when  operating  interacti vely,  and  interaction 
log  which  provides  a record  of  terminal  requests,  and  standard  output  reports. 

Oata  Base  Maintenance.  This  is  a collection  of  programs  required  to  maintain  the 
ETAM  master  files.  They  provide  the  facilities  for  initial  file  loading,  limited 
editing,  and  for  printer  or  terminal  dump  of  the  file  contents.  A similar  set  of 
maintenance  routines  will  also  be  required  for  the  ETAM  individual  project  files. 

ETAM  WORKING  DATA  BASE.  There  are  three  major  sections  of  this  data  base.  Specific 
developmental  implementation  of  these  functional  data  base  requirements  are  included 
in  Section  III  of  this  report. 

Program  Library.  The  first  section  of  the  ETAM  Working  Data  Base  is  a program 
library  containing  the  major  programs  and  routines  to  be  called  during  the  pro- 
cessing operations.  The  programs  contained  within  this  section  of  the  data  base 
will  be: 

1.  Description  Structuring 

2.  Range-of-Effect 

3.  Assessment 

4.  User  Interactive 

5.  Data  Base  Maintenance 

A direct  access  storage  device  will  be  used  for  storing  all  programs,  routines, 
and  subroutines  which  are  a part  of  the  program  library,  as  well  as  for  storing 
project  data,  conversion  tables,  abbreviated  records  of  entities  for  seach  re- 
trieval, and  extracted  data  from  Navy  data  bases. 
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Project  File.  The  second  major  section  of  the  ETAM  Working  Data  Base  will  contain 
all  project  data  loaded  initially  using  the  Description  Structuring  programs,  as  well 
as  any  additional  data  developed  as  a result  of  performing  the  ETAM  operations. 

These  data  will  include: 

1.  Project  Descriptive  Data  (from  Task  1) 

2.  Alternatives  to  the  Innovation  (from  Task  2) 

3.  Risk  Factors  and  Risk  Reduction  Projects  (from  Task  3) 

4.  Decision  Tree  Factors  (from  Tasks  4 and  6) 

5.  Range-of-Effect  Search  Descriptors  (from  Task  5) 

6.  Benefits  Patterns  (from  Task  5) 

7.  Cost-Benefits  Results  (from  Task  6) 

B.  Financial  Analysis  Data  (from  Task  7) 

Exterior  to  the  project  files,  abbreviated  records  of  all  courses,  jobs,  and  equip- 
ment will  be  maintained  as  a part  of  the  ETAM  Working  Data  Base.  These  records  will 
be  appropriately  indexed  to  permit  search  once  search  descriptors  have  been  formulated. 

Project  data  will  utilize  a non-hierarchical  structure  with  fixed  length  records. 
Relationships  between  primary  project  data,  alternatives  to  the  primary  project,  risk 
reduction  project  data,  and  entities  being  costed  and  aggregated  will  be  maintained 
through  cross-reference  information  contained  within  certain  project  file  types.  The 
components  of  the  project  file  are  defined  in  Section  III  of  this  report.  The  initial 
ETAM  procedures  served  as  a guide  in  development  of  project  descriptive  data  and  risk 
reduction  project  data. 

Extracted  Data.  The  third  section  of  the  ETAM  Working  Data  Base  serves  as  a repository 
for  the  full  set  of  data  extracted  from  searches  of  master  Navy  data  bases.  These 
data  will  be  maintained  throughout  the  assessment  operation  and  finally  summarized 
and  stored  as  a part  of  the  project  data.  Individual  data  sets  will  be  established 
for  entities  extracted  from  existing  Navy  data  bases.  Because  of  many  of  the  attributes 
of  courses,  job,  and  equipments  are  subject  to  frequent  change,  once  extracted  they 
will  be  used  for  processing  in  relation  to  a single  proposed  innovation  being  evaluated. 
The  entity  will  be  repeated  in  a separate  data  set  if  it  is  related  to  more  than  one 
project.  New  extraction  may  be  required  if  the  analysis  spans  a substantial  time 
period  or  if  the  entities  are  related  to  a major  new  program  which  is  being  subjected 
to  frequent  change. 

MASTER  DATA  BASE.  This  set  of  abbreviated  data  files  is  accessible  by  all  projects 
and  will  contain: 

Course  Abbreviated  Data  Base  Structures.  The  primary  use  of  the  abbreviated  data 
bases  is  for  searching  out  entities  which  match  the  descriptors  formulated  through 
the  assessor's  analysis  of  the  target  application  and  interactive  prompting  using  the 
taxonomies  related  to  the  entity  class  (e.g.,  courses).  Figure  1 1 -3  shows  a recommended 
structure  for  courses  to  be  maintained  with  the  project  file  area  of  the  ETAM  Working 
Data  Base. 
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FIELD  NAMES 

COP 

CIN 

CST 

NOBC 

NEC 

PC 

RMS 

TYCRS 

SVC 

MI 

STCD 

STDTE 

TRAPS 

TPC 

ATTR 

STBK 

CLEN 

CHRS 

CAOB 


DESCRIPTION 

Course  CDP  Number 

Course  Identification  Number 

Course  Short  Title 

Naval  Officer  Billet  Code 

Primary  NEC  Code 

Priority  Code 

Low-Order  3 Digits  of  RMS  Code 

Type  Course 

Service 

Method  of  Instruction 

Status  Code 

Status  Date 

TRAPS  Indicator 

Training  Program  Coordinator 

Attrition  Rate 

Setback  Rate 

Course  Length 

Total  Contact  Hours 

Cost  per  AOB 


FIGURE  1 1-3.  SAMPLE  COURSE  ABBREVIATED  DATA  BASE  CONTENTS 
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Job  Abbreviated  Data  Base  Structure.  As  with  the  course  abbreviated  data  base, 
the  job  abbreviated  data  base  will  be  used  to  search  job/task  entities  which  match 
descriptors  formulated  from  the  taxonomies  for  the  job/task  entity  class.  A rec- 
ommended structure  is  outlined  in  Figure  1 1-4 . 

Equipment  (Vehicle  Type)  Abbreviated  Data  Base  Structure.  The  inventory  of  equip- 
ments  used  within  naval  training  would  be  maintained  in  the  third  abbreviated  data 
base.  Figure  1 1-5  shows  a recommended  structure. 
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FIELD  NAMES  DESCRIPTION 

RATE  Rating/Pay  Grade 

JTASK  Job/Task  Code 

JNAME  Job/Task  Name 

CBIL  Cost  per  Billet 

FIGURE  1 1-4.  SAMPLE  JOB  ABBREVIATED  DATA  BASE  CONTENTS 


FIELD  NAMES 

SNUM 

DDEG 

DNAME 

DCOST 


DESCRIPTION 

Federal  Stock  Number 
Device  Designator  Number 
Device  Name 
Device  Cost 


FIGURE  1 1-5.  SAMPLE  TRAINING  VEHICLE  ABBREVIATED  DATA 
BASE  CONTENTS 
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SECTION  III 
ETAM  DEVELOPMENT 


This  section  reports  the  achievements  of  the  ETAM  Phase  II-B  effort.  Because 
full  reports  on  the  studies  on  Scaling  and  Indexing  and  the  document  on 
Innovations  in  Training  are  available  separately,  their  contents  will  be 
merely  summarized  here. 

PRELIMINARY  STUDIES 

SCALING  PROCEDURES.  The  objective  of  this  study  was  to  develop  procedures 
that  were  simple  but  psychologically  reliabe  and  valid  for  enabling  an  informed 
decision  maker  (or  the  staff  supporting  the  decision  maker)  to  express  the 
combined  subjective  value  scales  in  order  to  reach  a single  continuum  of 
worth  or  "utility"  on  which  to  set  and  evaluate  outcome  alternatives.  Real 
life  decisions  must  often  take  into  account  more  or  less  intangible  variables 
such  as  "mission  success,"  or  "increased  motivation,"  or  "increased  capacity 
for  initiative."  Attempts  to  convert  these  variables  into  objective  dollar 
data,  while  useful,  are  inevitably  incomplete  expressions  of  the  benefit  (or 
liability)  factors  inherent  in  the  concept  of  the  variaDle  in  real  world 
enterprise. 

The  study  had  to  depend  on  the  interpretation  of  the  state-of-the-art  in 
decision  theory  and  behavioral  utility  theory  rather  than  on  independent 
empirical  research.  The  activities  therefore  consisted  of  intensive  search 
and  review  of  an  extensive  body  literature.  Much  of  the  literature  is 
theoretically  oriented  towards  mathematical  models  attempting  to  simulate 
human  decision  processes.  Some  of  it  is  empirically  and  practically  oriented 
towards  giving  the  human  decision  maker  better  tools  tnat  minimize  niceties 
that  have  little  or  no  practical  value  but  greatly  complicate  the  mechanics 
of  the  procedure. 

Rationales  were  developed  for  decision  making  in  the  ETAM  context.  These 
included  a novel  concept,  the  idea  of  a "system  utility"  continuum  for  the 
executive.  The  anchor  point  of  this  continuum  was  in  the  present.  Another 
reference  point  would  be  the  idealized  level  of  the  variable.  Still  another 
would  be  the  projection  of  the  location  of  the  point  on  the  variable  into 
future  states  of  the  system,  assuming  no  change  in  the  process  but  changes  in 
the  system  environment.  The  projected  value  of  the  innovation's  contribution 
to  the  variable  could  then  be  located  with  respect  to  these  reference  points 
and  evaluated.  This  concept  shifts  the  focus  of  the  decision  maker  from  the 
classical  self-serving  basis  for  utility  and  choice  into  an  enterprise- 
serving reference. 

These  and  related  issues  became  part  of  the  instructional  content  in  the 
study  report,  and  are  intended  to  supplement  the  procedural  mechanics  for 
utility  scaling. 

A set  of  scaling  procedures  consistent  with  ETAM  requirements  was  developed. 

It  can  be  performed  manually,  with  paper  formats,  but  more  effectively  with 
computer  interaction.  The  user  is  relieved  of  a variety  of  paper-shuffling 
and  computational  mechanics,  and  can  readily  be  refreshed  with  source  data 
for  making  scaling  decisions. 
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A major  finding,  almost  a consensus,  in  the  practical  literature  is  that  the 
simplest  scaling  procedures  are  as  effective  as  the  more  complex.  This 
finding  was  reflected  in  the  ETAM  procedure. 

In  order  to  maintain  a common  sense  reference,  as  well  as  to  be  responsive  to 
individual  differences  in  assessors  and  decision  maker  preferences,  the  ETAM 
procedure  calls  for  the  global  or  wholistic  (intuitive)  method  to  be  used  as 
a check  against  the  results  of  the  "decomposition"  or  analytic  method. 

The  final  report  on  this  study  contains  a step-by-step  method  for  the  scaling 
operations  that  establish  the  inputs  for  the  decision  tree  content.  The 
report  also  summarizes  the  literature  that  was  examined,  and  contains  instruc- 
tional rationales  for  the  user's  orientation  to  decision  making  in  the  setting 
of  an  enterprise.  It  is  not  necessary  that  the  user  accept  the  premises  and 
rationales,  but  at  least  there  is  a reference  for  departure  from  them. 

The  study  personnel  undertook  some  informal  sample  tests  of  the  procedure  in 
some  mutual  and  simulated  situations.  It  was  found  that  the  procedures  are 
far  easier  to  perform  than  to  describe--mul ti variate  scaling  problems  could 
be  solved  in  ten  to  twenty  minutes.  The  results  conformed  to  retrospective 
common  sense;  that  is,  common  sense  applied  after  the  procedure  had  yielded 
results.  But  the  procedure  took  into  account  more  information  available  to 
the  user  than  appeared  to  be  used  in  global  or  intuitional  methods. 

These  are  not  definitive  tests,  of  course.  They  are  cited  primarily  to 
indicate  that  the  procedures  are  not  cumbersome  to  apply,  despite  the  appear- 
ance of  the  formal  presentation  of  the  method. 

INDEXING.  An  innovation  or  change  may  affect  perhaps  some  100  out  of  the 
over  4,000  courses  taught  by  the  Navy,  or  some  half-dozen  of  the  hundreds  of 
Navy  instructional  vehicle  types;  or  several  hundred  job-tasks  in  the  tens  of 
thousands  of  job-tasks  scattered  across  Navy  rates  and  ratings.  The  hard  way 
of  finding  the  targets  would  be  to  read  through  the  entire  catalog  of  course 
descriptions  in  Catalog  of  Navy  Training  Courses  (CANTRAC),  or  the  huge 
Occupational  Standards  directory  and  Navy  Occupational  Task  Analysis  Program 
(N0TAP)  listings,  or  several  thousand  pages  of  descriptions  of  the  Navy's 
instructional  devices. 

The  easier  way  would  be  to  attach  subject  matter  descriptors  to  each  of  the 
entities  in  these  inventories  of  items;  index  the  particular  innovation  from 
the  same  set  of  descriptors,  and  use  this  index  as  a search  argument;  then, 
enter  the  search  argument  into  a computerized  search  for  matching  entities. 
This  would  yield  at  least  a preliminary  set  of  candidate  entities  relevant  to 
the  innovation.  Final  decisions  of  relevance  would  require  the  human  to  scan 
the  context  information  about  the  entity  such  as  job-task  titles  and  descrip- 
tions, course  descriptions,  or  instructional  vehicle  descriptions.  But  the 
latter  task  could  be  confined  to  a manageable  subset  of  the  total  inventory. 

The  first  phase  of  ETAM  developed  a taxonomic  structure  for  job-tasks,  course 
objectives,  and  instructional  vehicles.  This  classification  aimed  at  identi- 
fying essential  attributes  of  the  teaching-learning  process  and  its  supporting 
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elements.  One  objective  of  the  present  study  was  to  try  out  this  classifica- 
tion scheme  by  indexing  samples  of  each  of  the  three  inventory  types  (courses, 
job-tasks,  vehicles)  to  determine  indexing  feasibility.  The  second  phase  of 
this  demonstration  was  to  create  a data  base  structure  for  automatic  search 
with  sample  descriptor-indexed  innovations.  A second  objective  was  to  formu- 
late overall  strategies,  not  limited  to  the  automatic  phase  of  convergence, 
in  determining  the  full  range-of-effect  of  a proposed  innovation.  These 
strategies  of  necessity  would  have  to  compromise  the  ideal  with  the  practical. 
The  third  objective,  growing  out  of  the  first,  was  to  demonstrate  at  least  a 
limited  tryout  with  an  interactive  computer  procedure  for  entering  a search 
specification  and  receiving  processed  output.  The  results  of  the  study  con- 
sisted of  a report  on  recommended  indexing  procedures  and  search  procedures 
and  demonstrations  of  interactive  computer  processing.  (See  Appendix  A.) 

A paper  format  was  prepared  for  indexing  training  courses  from  CANTRAC  course 
descriptions.  The  total  descriptor  set  was  about  50  terms,  grouped  into  less 
than  a dozen  categories.  They  were  taken  directly  from  the  ETAM  taxonomy. 

The  indexer  was  highly  familiar  with  the  meanings  of  these  descriptors  and 
had  some  general  background  in  the  content  of  training  courses.  After  several 
hours  of  practice,  the  indexing  task,  although  difficult,  was  not  overwhelming. 
Up  to  15  courses  could  be  indexed  in  an  hour.  This  pace  could  accelerate 
when  a given  type  of  course  could  be  identified  and  matched  with  a fairly 
standard  pattern  of  descriptors.  The  same  experience  applied  to  indexing 
job-task  titles  in  the  context  of  rates  and  ratings  in  the  Occupational 
Standards  directory,  except  that  a faster  pace  could  be  maintained.  The  more 
specific  the  description  and  the  clearer  the  context,  the  faster  and  more 
certain  the  indexing  activity  can  be.  The  quality  of  descriptions  of  training 
devices  and  their  application  was  highly  variable,  and  this  created  greater 
uncertainty  in  many  cases  in  the  selection  of  applicable  descriptors. 

A key  factor  that  enabled  coping  with  ambiguity  in  selecting  descriptors  for 
an  entity  was  recognizing  that  the  final  selection  of  relevant  entities  would 
not  be  restricted  to  automatic  search,  but  to  human  examination  of  contextual 
information  about  the  entity.  This  means  that  automatic  search  should  result 
only  in  excluding  from  further  examination  those  entities  with  high  probability 
of  low  yield.  The  indexer's  policy  therefore  was,  when  ambivalent  whether  or 
not  to  apply  the  descriptor  to  the  entity,  to  apply  it.  This  may  later  add 
an  increment  of  labor  to  the  human  in  culling  it  it  out  manually.  But  if  the 
descriptor  was,  for  later  range-of-effect  determinations,  applicable,  the 
entity  would  not  be  lost  for  inclusion.  This  policy  was  also  incorporated 
into  recommendations  for  indexing  the  innovation. 

The  human  assessor  may  choose  to  bypass  the  ETAM  level  of  search  and  go 
directly  to  the  applicable  Navy  data  base  of  courses,  job-tasks  or  instruc- 
tional vehicles.  The  coding  of  job-task  titles  in  the  Occupational  Standards 
embodies  a secondary  subject  matter  index.  Occupational  Standards  listings 
are  computerized,  and  their  contents  can  be  selectively  retrieved,  some 
interactively.  The  listings  are  also  tied  into  the  computerized  NOTAP  data 
which  provide  extensive  context  information  about  each  job-task  listed  in  the 
Occupational  Standards.  Whereas  ETAM  descriptors  are  limited  primarily  to 
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task  "functions,"  the  NOTAP  descriptions  include  tools  and  equipment  used, 
importance  and  frequency  of  tne  job- task,  ana  other  significant  details. 

They  also  reference  the  Navy  courses  where  the  requisite  skill  is  trained. 

With  very  minor  modifications,  and  within  the  limited  range  of  practical 
testing  applied  thus  far,  the  ["AM  taxonomy  adequately  serves  the  indexing 
and  search  functions.  The  fact  that  the  descriptor  set  applied  to  each  of 
the  three  classes  of  entities  is  restricted  to  about  50  items,  grouped  into 
meaningful  subsets,  is  a substantial  advantage--the  entire  set  can  be  dis- 
played on  a single  page  and  scanned  in  a few  moments. 

REPORT  ON  MAJOR  INNOVATIONS  IN  TRAINING  TECHNOLOGY.  The  objective  of  this 
report  was  a readable  and  interesting  document  about  changes  in  training 
technology  during  the  past  25  years.  The  target  audience  consisted  of  exe- 
cutives and  their  staff  in  the  field  of  education  and  training,  as  well  as 
specialists  with  a need  for  overviews  and  background.  Emphasis  was  to  be 
placed  on  concepts  and  doctrines  underlying  the  innovations  as  well  as  the 
essential  features  of  the  innovations  themselves.  It  was  to  be  a comprehen- 
sive overview  rather  than  a detailed  catalog.  The  preparation  of  the  report 
was  a collaborative  effort  between  Dr.  Alfred  F.  Smode  of  the  Navy  Training 
Analysis  and  Evaluation  Group  and  Dr.  Robert  B.  Miller  of  IBM. 

The  content  of  the  document  is  not  readily  summarized  in  a few  paragraphs, 
but  several  major  topics  can  be  identified.  An  evolutionary  change  is  shift- 
ing training  doctrine  from  teaching  about  a subject  matter  to  the  learning  of 
skills  and  competences.  Information  about  the  learning  process  and  about 
individual  differences  becomes  integrated  into  "learner-centered"  instruction. 
Understanding  about  tasks  and  skills  is  reflected  in  clearer  definitions  of 
training  objectives  and  performance  criteria,  with  improved  payoff.  Models 
of  tradeoff  variables  in  training  and  learning--such  as  motivation  and  native 
ability  as  examples--permit  greater  flexibility  and  more  economical  practices. 
New  social  and  psychological  models  of  work  and  the  worker  can  and  should  be 
reflected  in  the  conditions  of  learning. 

A tighter  and  therefore  more  economical  coupling  is  being  developed  in  the 
sequence  of  technical  activities  that  begins  with  the  need  for  a new  skill; 
the  identification  of  the  components  of  that  skill;  the  preparation  of  means 
for  selecting,  training  and  evaluating  the  learner;  and  his  assignment  and 
evaluation  on  the  job. 

The  computer  is  playing  important  and  diverse  roles.  It  is  central  to  the 
control  of  wide  ranges  in  type  of  instruction  from  the  format  of  programmed 
instruction  to  the  realistic  simulation  of  entire  aircraft  missions  from 
startup  to  shutdown.  It  enables,  at  a cost,  the  highly  realistic  performance 
of  the  properties  of  the  vehicle  operating  under  the  management  of  the  student 
operator  in  a wide  range  of  environments.  But  in  addition,  the  computer  is  a 
powerful  extension  of  the  instructor  in  measuring  and  evaluating  behavior,  in 
changing  the  level  or  kind  of  problem  the  student  has  to  solve,  and  in  giving 
guidance.  The  instructor  becomes  a manager  of  instruction  rather  than  a 
mechanic  who  manipulates  the  devices  of  instruction.  But  the  training  capa- 
bility of  the  device  is  nonetheless  limited  to  the  instructional  purposes  and 
designs  to  which  it  is  put,  and  these  depend  on  human  insight  and  competence. 
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The  Navy's  training  organization  is  a vast  enterprise,  with  inventories  of 
thousands  of  courses,  instructors,  students,  devices,  and  other  resources. 

The  control  and  management  of  this  enterprise  requires  increasing  dependence 
on  the  computer  and  computer  models  for  scheduling  resources  in  coping  with 
complex  and  changing  demand  patterns  and  varied  pipeline  inputs.  Both 
budgetary  and  time  constraints  demand  effectiveness  in  operation  with  high 
levels  of  efficiency.  But  computer  operations  demand  a variety  of  standardi- 
zations in  operation  that  may  conflict  with  flexibility  for  the  needs  at  the 
moment.  The  significant  innovation  is  the  interactive  computer  that  enables 
the  human  to  intervene  in  the  operation  or  decision  which  is  standard  for  the 
computer  program  and  to  modify  its  output,  but  doing  so  with  a display  of  the 
consequences  of  his  action  before  they  actually  occur.  The  computer  program, 
thus,  becomes  a tool  to  human  judgment  ’n  the  situation  of  the  moment. 

Telecommunications  around  the  world  make  it  increasingly  feasible  to  put 
training  where  the  student  happens  to  be  rather  than  having  to  bring  him  to 
the  classroom.  This  capability  will  be  supported  by  decreasing  costs  and 
miniaturization  of  computers.  It  can  be  anticipated  that  learning  and  doing 
will  become  more  intimately  related  in  the  job  environment  itself.  This  will 
be  another  innovation  made  possible  by  engineering  technology  and  by  possibly 
radical  changes  in  the  concept  of  job  competence  and  its  relationship  to  the 
learning-training  process. 

The  foregoing  are  samples  of  some  ideas  that  thread  through  the  document.  The 
structure  of  the  report  contains  several  levels  of  summarization.  An  early 
chapter  embeds  one  level  of  summarization.  Each  chapter  has  its  own  summary. 

ETAM  PROGRAM  FLOW 

ETAM  consists  of  a sequence  of  data  collection,  data  input,  data  analysis, 
interactive  data  refinement,  computational,  and  data  output  steps.  The 
system  is  designed  to  perform  on  the  National  CSS  timesharing  system.  The 
user  interacts  with  the  system  in  three  basic  ways. 

i \ 

1.  ETAM  is  run  interactively  with  the  user  specifying  certain  pro- 
cessing sequences,  data  inputs,  and  output  requests  in  a real-time 
mode. 

2.  User  project  files  are  created  or  updated  in  batch  mode  via  card 
input  shipped  to  a National  CSS  processing  center;  e.g.,  Atlanta, 
Georgia.  Similarly,  the  batch  mode  of  operation  can  be  used  for 
processing  the  more  voluminous  output  reports  which  will  be  run  at 
the  National  CSS  processing  center  and  shipped  to  the  user. 

3.  System  data  bases  accessible  to  each  user  (and  project)  are  generated, 
updated,  or  printed  through  system  generation/data  base  maintenance 
routines  which  are  part  of  the  main  ETAM  process  flow. 

This  section  describes  the  major  data  sources  which  support  the  ETAM  processing 
sequence.  The  basic  model  processing  steps  are  also  described.  Programming 
development  has  been  completed  through  the  Range-of-Effect  part  of  the  ETAM 
processing  sequence,  therefore,  the  data  files  and  processing  steps  described 
for  Range-of-Effect  reflect  the  operational  system.  The  Assessment  part  of 
the  ETAM  processing  sequence  has  been  completed  through  the  program  design 
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level,  therefore,  the  documentation  of  that  process  provides  detailed  speci- 
fications and  design  guidance  for  completing  the  ETAM  program  development. 
Figure  III-l  shows  the  ETAM  project  data  base  files,  programs,  and  their 
interrelationships. 

PROJECT  DATA  BASE  FILES.  The  Project  Data  Rase  (PDB)  stores  all  initially 
entered,  prompted,  and  calculated  information  for  each  project  being  evaluated 
by  ETAM.  Each  project  is  assigned  a unique  file  name  (fn);  e.g.,  fn  = PN123, 
and  may  have  up  to  twenty-two  file  types  (ft);  e.g.,  ft  = ID,  ft  = SCEN, 
etc.,  associated  with  each  fn.  For  example,  the  fn  ft  called  PN123  ID  will 
store  the  descriptive  text  for  project  PN123.  The  twenty-two  file  types 
which  have  been  identified  for  each  project  are  as  follows: 

ID  (Identification).  This  file  contains  descriptive  information  on  the 
project  such  as  Project  Name,  Abstract,  Objectives,  Experimental  Results,  and 
Target  Applications.  The  data  are  stored  in  80-column  card  image  format  with 
major  paragraph  headings  as  identified  in  the  examples  above.  Blank  cards 
may  be  inserted  freely  to  improve  readability  of  output  reports  containing 
exact  images  of  this  file.  The  first  card  in  this  file  must  contain  the  file 
name;  i.e.,  PNnnn  anywhere  in  the  first  ten  columns.  Columns  11-80  will 
contain  the  Project  Name.  An  example  of  the  ID  file  content  is  shown  in 
Figure  1 1 1 - 2 . It  can  be  printed  with  the  PI  print  program. 

REC  (Range-of-Effect,  Course).  An  innovation  which  has  potential  application 
to  Navy  courses  will  be  described  with  one  or  more  course-rel atea  descrip- 
tors. This  file  contains  initial  descriptors  assigned  by  the  assessor,  or 
interim  or  finalized  descriptors  resulting  from  possibly  several  iterations 
of  prompting,  course  searches,  and  analysis.  Each  descriptor  number,  with  an 
associated  sign,  is  placed  anywhere  in  the  first  ten  position  of  a card-image 
field.  Descriptor  names  are  not  required  as  they  will  be  inserted  from  a 
Course  Descriptor  file  during  processing.  The  negative  descriptor  number  is 
allowed  to  perform  a NOT  logic  function  during  the  search  operation.  An 
example  of  REC  file  content  is  shown  in  Figure  1 1 1 -3 . 

REV  (Range-of-Effect,  Vehicle).  Instructional  vehicles  may  be  affected  by 
certain  innovations.  The  innovation  will  then  be  described  with  vehicle- 
related  descriptors.  Format  and  use  are  the  same  as  the  REC  file.  Descrip- 
tor names  will  be  obtained  during  processing  from  the  Vehicle  Descriptor 
file. 

REJ  (Range-of-Effect,  Job/Tasks).  Innovations  which  may  affect  certain 
job/tasks  within  the  Navy  will  be  described  with  job-related  descriptors. 
Format  and  use  of  this  file  is  the  same  as  REC.  Again,  descriptor  names  will 
be  obtained  from  a Job/Task  Descriptor  file  and  inserted  during  process. ng. 

SARG  (Search  Arguments).  This  is  a non-printable , internally  generated  file 
which  provides  a map  of  the  REC,  REV,  and  REJ  descriptors  for  subsequent 
search  of  the  Abbreviated  Data  Bases  for  Courses,  Vehicles,  and  Job/Tasks. 

The  file  contains  three  240-character  records.  The  first  record  contains  a 
map  of  the  REC(Course)  descriptors,  two  the  REV(Vehicle)  descriptors,  and 
three  the  REJ(Job/Task)  descriptors.  The  format  of  the  SARG  file  is  as 
fol lows: 
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♦♦ID 

PN123  3-D  PROCEDURAL  TRAINER 


OBJECTIVES  AND  SUMMARY  DESCRIPTION 

THE  OBJECTIVE  IS  TO  INCREASE  THE  EFFICIENCY  AND  KETENTIVITY  OF  THE 
LEARNING  OF  OPERATING  PROCEDURES  ON  ELECTRONIC  EQUIPMENT.  THE  DEVICE 
CONSISTS  OF  THE  SHELL  OF  THE  ELECTRONIC  GEAR  ON  WHICH  THE  PROCEDURES 
ARE  TO  BE  LEARNED.  WITHIN  THIS  SHELL,  SPECIAL  CIRCUITS  ARE  HOOKED  UP 
TO  A COMPUTER  WHICH  SIMULATES  THE  BEHAVIOR  OF  THE  EQUIPMENT  TO  THE 
ACTIONS  OF  THE  STUDENT,  BUT  ALSO  PROVIDES  CORRECTIVE  AND  OTHER  INSTRUCTIONAL 
GUIDANCE.  THE  COMPUTER  IS  PROGRAMMABLE  SO  AS  TO  ADAPT  TO  THE  LEARNING 
OBJECTIVES. 

DELIVERY  SYSTEM 

THE  INNOVATION  CENTERS  AROUND  A DELIVERY  SYSTEM  FOR  PROCEDURES  LEARNING. 

THE  TRAINER  DISPLACES  THE  OPERATIONAL  LIVE  EQUIPMENT  THAT  IS  USED  FOR 
INSTRUCTIONAL  PURPOSES  AND  THE  INSTRUCTIONAL  MANUALS  THAT  GUIDE  THE 
STUDENT  IN  HIS  LEARNING  THE  PROCEDURE. 

INSTRUCTIONAL  PROCESS 

THE  DEVICE  ENABLES  THE  STUDENT  TO  PERFORM  MANUAL  PROCEDURES  WITH  ACTUAL 
DISPLAYS  AND  CONTROLS  AND  THEIR  RELATIONSHIPS.  HE  ALSO  GETS  INSTRUCTIONAL 
FEEDBACK  AND  GUIDANCE  FROM  AN  AUXILIARY  DISPLAY  SCREEN  CONTROLLED  BY  THE 
COMPUTER.  THE  STUDENT  ACTUALLY  LEARNS  BY  DOING  THE  TASK,  WITH  SUPPLEMENTAL 
INDIVIDUALIZED  HELP  AND  DIRECTION  CONTROLLED  BY  THE  RESPONSES  HE  HAS  ALREADY 
MADE.  AND  SHOULD  BE  MAKING  NEXT.  THIS  SHOULD  RESULT  IN  FASTER  LEARNING, 
BETTER  REMEMBERING,  AND  HIGHER  PROCEDURAL  LEVEL  SKILL.  THE  DEVICE  IS  A 
FORM  OF  SIMULATOR. 

JOBS  AND  TASKS 

OF  PRIMARY  RELEVANCE  ARE  THOSE  JOBS  THAT  DEAL  WITH  THE  EXTERNAL  DISPLAYS 
AND  CONTROLS  OF  INTACT  ELECTRONIC  EQUIPMENT.  THIS  MAY  INCLUDE  TESf  EQUIPMENT 
AND  OPERATOR  TRAINING  IN  SEQUENTIAL  PROCEDURES  ON  THE  EQUIPMENT  SUCH  AS  TURN 
ON,  START  UP,  CHECKOUT,  AND  OPERATION.  ALSO,  MAINTENANCE  TASKS  DONE  ON  THE 
OUTSIDE  OF  THE  INTACT  ELECTRONIC  EQUIPMENT  SUCH  AS  CHECKOUT,  ALIGNMENT  ETC. 

EQUIPMENT  ORIENTATION 

ELECTRONIC  EQUIPMENT.  NO  RESTRICTIONS  ON  EQUIPMENT  TYPE. 

WORK  FUNCTION 

SEQUENTIAL  PROCEDURES:  READING  INDICATORS;  ACTIVATING  CONTROLS  AS 

PRESCRIBED  IN  SEQUENCE;  RECOGNIZING  DEVIATIONS  FROM  "NORMAL"  AND  TAKING 
SPECIFIED  ACTION. 

OPERATIONAL  ENVIRONMENT 

ANY  LAB  ENVIRONMENT:  NO  PHYSICAL  MOTION;  NORMAL  TEMPERATURES.  THE 

STUDENT  IS  UNSTRESSED  EXCEPT  AS  MAY  BE  ARTIFICIALLY  INDUCED  BY  VERBAL 
MEANS.  SPACE  USED  BY  THE  TRAINER  IS  PRACTICALLY  EQUIVALENT  TO  THAT  USED 
BY  OPERATIONAL  EQUIPMENT  FOR  TRAINING. 

STUDENT  ATTRIBUTES 

MAY  HELP  SLOW  LEARNERS  OF  ROUTINE  TASKS.  STUDENT  ATTRIBUTE  FACTORS  AR  t NOT 
A MAJOR  TARGET  OF  THE  INNOVATION. 

PRIOR  TRAINING  AND  EXPERIENCE 

THE  TRAINER  COULD  BE  AN  EFFICIENT  WAY  OF  PROVIDING  REFRESHER  TRAINING, 

BUT  THIS  HAS  NOT  BEEN  INVESTIGATED  AS  A JUSTIFICATION. 

ATTITUDES  AND  MOTIVATIONS 

THESE  SHOULD  BE  IMPROVED  AS  A BY-PRODUCT  OF  THE  STUDENT  GETTING 
IMPERSONAL,  INDIVIDUAL,  AND  CONTINUOUS  PRACTICE  RATHER  THAN  WAITING  • 

AROUND  FOR  THE  INSTRUCTOR'S  ATTENTION,  OR  MAKING  MISTAKES  AND  NOT  KNOWING 
WHAT  HE  DID  INCORRECTLY  FOR  SUBSTANTIAL  PERIOOS  OF  TIME.  THE  STUDENT  MAY, 
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EARLY  IN  LEARNING  THE  TASK,  GET  A PROPER  SENSE  OF  PACE  WHICH  CAN  BE  FORCED 
BY  THE  PROGRAMMING  OF  THE  INSTRUCTIONAL  GUIDANCE. 

APTITUDES 

THESE  MAY  BE  AFFECTED,  BUT  ONLY  AS  A BY-PRODUCT  OF  OTHER  FACTORS.  THE 
TRAINER  CAN  REDUCE  THE  DEPENDENCE  OF  INSTRUCTOR  AND  STUDENT  ON  THE  STUDENT  ’ S 
ABILITY  TO  LEARN  SYMBOLIC  MATERIALS  WHICH  MAY  NOT  BE  HIGHLY  CORRELATED 
WITH  HIS  ABILITY  TO  LEARN  THE  PROCEDURES  BY  ACTUAL  PHYSICAL  OPERATIONS. 

SOCIAL  FACTORS 

NO  INTENDED  INFLUENCE. 

SETBACKS  AND  ATTRITION 

THE  REDUCTION  OF  THESE  IS  A SECONDARY  TARGET. 

STUDENT  EVALUATION 

A CONTINUOUS,  OBJECTIVE  RECORD  OF  THE  ERRORS  MADE  BY  THE  STUDENT  IS 
MAINTAINED  THIS  SHOULD  BE  SUPERIOR  TO  INSTRUCTOR  EVALUATIONS.  THE 
DEVICE  MAY  BE  USED  TO  KEEP  THE  STUDENT  PRACTICING  UNTIL  THE  OESIRED 
PERFORMANCE  CRITERION  IS  REACHED. 

EMPIRICAL  OATA 

STUDY  PERFORMED  ON  TWO  PIECES  OF  EQUIPMENT. 

QUANTITATIVE  BENEFITS 
MEASURED  RESULTS 

CONTROL:  NORMAL  TRAINING  OPERATIONS. 

EXPER:  USE  OF  INNOVATION  ON  SAME  EQUIPMENTS,  SAME  PROCEDURES 

TO  BE  LEARNED. 

RESULTS:  TESTED  ON  LIVE  EQUIPMENT  AFTER  EQUAL  HOURS  OF  ELAPSED  TIME. 

1.  IMMEDIATELY  AFTER  TRAINING:  EXPER  WERE  UO%  SUPERIOR. 

2.  RETEST  ONE  MONTH  LATER:  EXPER  60%  SUPERIOR.  (EQUIVALENT 

TO  REDUCING  OJT  FROM  A WEEKS  TO  1 WEEK.) 

3.  TEST  SIX  MONTHS  AFTER  ON  JOB: 

EXPER:  30%  RATED  "OUTSTANDING." 

CONTROL:  5%  RATED  AS  "OUTSTANDING." 

A.  ATTRITION 

EXPER:  1? 

CONTROL:  7% 

CRITERION  VARIABLES  FOR  MEASURING  TRAINING  EFFECTIVENESS 
VALIDITY  AND  COMPLETENESS  IN  LEVEL  OF  PROFICIENCY. 

TARGET:  HIGHER  LEVEL  OF  OPERATIONAL  PROFICIENCY  IN  PROCEDURES 

PERFORMANCE. 

QUANTITATIVE  COSTS 

ESTIMATED  OPERATION  COSTS  FOR  INNOVATION,  8 YEARS:  $7.00  PER  STUDENT  HOUR. 
PRESENT  OPERATING  COST  FOR  TRAINING:  $8.50  PER  STUDENT  HOUR. 

EXPER:  ONE  INSTRUCTOR  PER  20  STUDENTS. 

CONTROL:  ONE  INSTRUCTOR  PER  A STUDENTS. 

ESTIMATES  OF  MAGNITUDES  OF  B ENEF I TS/ S A VI NGS  UNDER  ACTUAL  CONDITIONS 
80%  REDUCTION  IN  ATTRITION  RATE. 

75%  SAVINGS  IN  OJT  PER  INCUMBENT. 

REDUCTION  IN  STANDBY  AND  SPARE  EQUIPMENT. 

DECREASES  IN  MISSION  FAILURES. 

ESTIMATES  OF  THE  PRACTICAL  IMPORTANCE  OF  THE  TRAINING  PRODUCT 

PRACTICAL  PROCEDURAL  PROFICIENCY:  CRUCIAL  IMPORTANCE  TO  OPERATIONS. 

RESOURCE  REQUIREMENTS  FOR  FURTHER  STUDY 

TECHNICAL  DEVELOPMENT:  $12  MILLION  OVER  3 YEA.RS. 

EMPIRICAL  STUDIES  $ 3 MILLION  OVER  3 YEARS. 


FIGURE  III— 2.  ID  FILE  EXAMPLE  (Cont'd) 
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31 
40 
64 

-80 

-81 

83 

84 

85 

86 
87 
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Pos 


1-  8 Record  1 = COURSES 

2 = VEHICLES 

3 = TASKS 


Pos  9-  10  A fixed  binary  designation  of  the  number  of 
descriptors  which  will  be  available  as  search 
arguments. 

Pos  11-240  115  zeros  or  signed  fixed  binary  values  which 

have  a positional  relationship  to  the  descriptors 
for  courses,  vehicles,  or  job/tasks;  e.g.,  Pos 

11-12  corresponds  to  descriptor  01,  Pos  69-70 
to  descriptor  30,  etc. 


The  REC  file  shown  in  Figure  1 1 1-3  contained  the  following  search  or  query 
descriptors:  30  31  40  64  -80  -81  83  84  85  86  87.  An  example  of  the 
SARG  file  which  relates  to  the  contents  of  this  example  REC  file  is  shown  in 
Figure  1 1 1-4 . 

RESU  (Search  Results).  This  file  contains  the  entities  from  the  Abbreviated 
Data  Bases  which  had  descriptors  matching  the  search  arguments  in  the  SARG 
file.  The  search  results  can  subsequently  be  refined  by  adding  or  deleting 
entities. 


The  format  of  the  RESU  file  is  as  follows: 


Pos 

Pos 

Pos 

Pos 


1 

2 

3 

4- 


16 


Entity  Type;  i.e.,  Course(l),  Vehicle(2),  Job/Task(3) 
Record  Type;  i.e.,  Original (1),  Added(2),  De1eted(3) 
Status  of  Abbreviated  Data;  i.e.,  Avail(l),  Unavail(2) 
Identification  of  Entity;  i.e., 


Course(Type  1 ) 

Pos  4-  11 

Pos  12-15 
Pos  16 

Vehicle(Type  2) 
Pos  4-  16 

Job/Task(Type  3) 
Pos  4-  8 

Pos  9-  10 

Pos  11-  15 

Pos  16 


Course  Identification  Number(CIN) 
Course  Data  Processing(CDP) 

Blank 


Vehicle  ID 

Rating;  e.c 
Rank;  e.g. , 
Job/Task;  e 
Blank 


. , A0,  BM 

3,  CS 

.g.,  34410 


An  example  of  the  results  in  RESU  from  searching  the  Abbreviated  Data  Base 
for  Courses  using  the  contents  of  REC  is  shown  in  Figure  1 1 1 - 5 . 


The  RESU  file  can  be  printed  either  from  the  PI  print  program  or  from  P5. 

r 

REE  (Range-of-Effect,  Extract).  This  file  has  the  same  format  as  the  RESU 
file.  It  will  contain  all  search  results  except  the  entities  which  were 
subsequently  deleted  when  the  search  results  were  refined  interactively. 
Thus,  it  will  contain  only  Original (1)  and  Added(2)  record  types.  Its  pur- 
pose is  to  reflect  entities  for  which  additional  data  are  being  requested  to 
be  used  in  the  Assessment  process.  It,  therefore,  reflects  the  content  of 
the  printed  Extract  Reports,  except  the t the  Extract  Reports  will  contain 
additional  descriptive  data  obtained  from  the  Abbreviated  Data  Bases. 
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REC  POS 

RECORD  CONTENT 

DESC  NO. 

1-  8 

COURSES 

9-  10 

11 

11-  68 

All  plus  zeros(29) 

(01-29) 

69-  72 

+1+1 

(30-31) 

73-  88 

All  plus  zeros (8) 

(32-39) 

89-  90 

+1 

(40) 

91-136 

All  plus  zeros(23) 

(41-63) 

137-138 

+1 

(64) 

139-168 

All  plus  zeros(15) 

(65-79) 

169-172 

-1-1 

(80-81) 

173-174 

All  plus  zeros(2) 

(82) 

175-184 

+1+1+1+1+1 

(83-87) 

185-240 

All  plus  zeros(56) 

(88-115) 

FIGURE  I II -4  SARG  FILE  EXAMPLE 
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EXC  (Extract  Courses).  This  file  contains  the  additional  data  on  the  courses 
matched  in  the  Range-of-Effect  search.  These  data  are  required  to  run  the 
Training  Cost  Model  in  order  to  calculate  the  cost  effects  of  applying  the 
benefit  pattern,  associated  with  implementing  the  innovation,  to  the  range  of 
courses  which  will  benefit  from  the  innovation.  The  data  are  presently 
planned  to  be  collected  from  appropriate  Navy  sources.  Much  of  the  data  are 
available  in  NITRAS,  however,  other  data  must  be  collected  manually.  A list 
of  the  data  elements  required  for  each  Range-of-Effect  selected  course  is 
V shown  in  Figure  1 1 1-6. 

EXV  (Extract  Vehicles).  This  file  contains  the  additional  data  on  vehicles 
matched  in  the  Range-of-Effect  search.  These  data  are  required  to  input  the 
Vehicles  Cost  Model  in  order  to  calculate  the  effect  of  implementing  an 
innovation.  The  data  are  presently  planned  to  be  collected  from  appropriate 
Navy  sources;  e.g.,  the  Naval  Training  Equipment  Center,  Orlando,  Florida. 

A list  of  the  data  elements  required  for  each  Range-of-Effect  selected  vehicle 
is  shown  in  Figure  1 1 1 - 7 . 

EXJ  (Extract  Job/Tasks).  This  file  contains  the  additional  data  on  job/tasks 
matched  in  the  Range-of-Effect  search.  These  data  are  required  to  input  the 
job/task  cost  model  for  assessing  the  cost  pattern  in  the  job  domain  as  a 
result  of  accepting  and  successfully  implementing  the  proposed  innovation. 
These  data  are  presently  planned  to  be  collected  from  appropriate  Navy  sources 
e.g..  Bureau  of  Naval  Personnel,  Occupational  Standards  Development  and  Navy 
Occupational  Task  Analysis  Program.  A list  of  the  data  elements  required  for 
each  Range-of-Effect  selected  job/task  is  shown  in  Figure  III-8. 

EXP  (Extract,  Defaults).  This  file  contains  technical  and  cost  data  which 
should  be  used  by  the  three  cost  model  programs  when  data  are  not  available 
in  the  appropriate  extract  file;  i.e.,  EXC,  EXV,  or  EXJ.  For  example,  the 
input  variables  required  as  input  to  the  Training  Cost  Model  were  defined 
under  the  EXC  file.  The  model,  as  presently  designed  and  listed  in  Appendix 
C,  contains  built-in  default  values  to  substitute  in  the  program  when  the 
data  requested  in  the  READ  operation  are  missing.  Since  changes  in  default 
values  would  require  program  modification  and  recompilation,  when  the  final 
training  cost  model  is  developed  the  default  values  should  be  substituted 
with  variables  which  reference  the  EXD  file.  Thus,  as  default  values  change, 
only  this  file  will  require  updating.  Refer  to  Appendix  C,  Page  C-5  for  the 
variables  identified  in  the  Training  Cost  Model  for  which  default  values  will 
be  required. 

BPT  (Benefit  Pattern).  Each  innovation  being  assessed  by  ETAM  will  have  a 
pattern  of  benefits  which  can  be  described  and  stored  in  the  care-image  text 
file.  Benefit  variables,  both  quantitative  and  qualitative,  and  the  manner 
in  which  the  innovation  affects  them  should  be  commented  upon  in  this  file. 
This  descriptive  information  will  be  used  by  the  assessor  in  establishing  the 
model  inputs  interactively.  An  example  of  the  BPT  file  content  is  shown  in 
Figure  1 1 1 - 9 . 

CMR  (Cost  Model  Results).  The  outputs  of  the  three  cost  models;  i.e.,  Train- 
ing, Vehicles,  and  Job/Tasks,  will  be  stored  in  this  file  for  use  in  the 
Decision  Analysis  and  Financial  Analysis  routines. 
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REC  POS 

VARIABLE  DESCRIPTION 

SYMBOL 

LENGTH 

FORMAT 

1-4 

Course  Number  (CDP) 

CDP 

4 

A4 

5-6 

Planning  Period  (Years) 

N 

2 

12 

7-9 

Attrition  Rate  (PCT) 

ARATE 

3 

F3.2 

10-13 

Training  Length  (Weeks) 

TLENGH 

4 

F4.1 

14-15 

Recycle  Rate 

RCRATE 

2 

F2.2 

16-18 

Ave  Recycle  Time  (Weeks) 

ARCYTM 

3 

F3.1 

19-21 

Weeks  School  Operates 

WSCHOP 

3 

F3.1 

22-23 

Time  Student  Position  Down  (PCT) 

TSPOSD 

2 

F2.2 

24-27 

Instr/Student  Positive  Ratio 

INTSPO 

4 

F4.2 

28-30 

Admin/Student  Position  Ratio 

AMTSPO 

3 

F3.3 

31-33 

Equip/Student  Position  Ratio 

EQUSPO 

3 

F3.2 

34 

Purchase  Policy 

PURCHF 

1 

11 

35 

Depreciation  Policy 

DEPF 

1 

11 

36-37 

Life  of  Equipment 

LOFEQ 

2 

12 

38-42 

SQ  Feet/Student  Position 

SQFTST 

5 

F5.2 

43-47 

SQ  Feet/Instructor  Position 

SQFTIN 

5 

F5.2 

48-52 

SQ  Feet/Admin  Position 

SQFTAM 

5 

F5.1 

53-56 

Course  Dev  Hours/Course  Hour 

IMDDEV 

4 

F4.1 

57-59 

Percent  Course  Requiring  Dev  (PCT) 

PUIMD 

3 

F3.2 

60-61 

Percent  Instr  Mat  Maintained  (PCT) 

UPDATE 

2 

F2.2 

62-65 

Supplies/Student  Position 

SUPSPO 

4 

F4.1 

66-69 

Supplies/Student 

SUPSTD 

4 

F4.1 

70-73 

Misc/Student  Position 

MSCSPO 

4 

F4.1 

74-77 

Misc/Student 

MSCSTD 

4 

F4.1 

78-79 

Percent  Extra  Student  Positions  (PCT) 

PESP 

2 

F2.2 

80-83 

Time  in  Medium  (Hrs/Wk) 

TLEGTH 

4 

F4.1 

84-85 

Life  of  Facility 

LOFFA 

2 

12 

86-88 

Week  School  Available 

WSH0P1 

3 

F3.1 

89-95 

Student  Salary  (Annual) 

STUDSL 

7 

F7.2 

96-101 

Student  Travel  Cost  To/From 

STCST1 

6 

F6.2 

102-107 

Student  Travel  Cost  in  Course 

STCST2 

6 

F6.2 

108-114 

Instructor  Salary  (Annual) 

INSTSL 

7 

F7.2 

115-121 

Admin  Salary  (Annual) 

ADMSL 

7 

F7.2 

122-129 

Equipment  Unit  Cost 

EQUNTC 

8 

F8.2 

130-132 

Pet  Max  Equip  Purchased 

PCTPCH 

3 

F3.2 

133-137 

Facility  Cost/SQ  FT 

CPSQFT 

5 

F5.2 

138-142 

Cost/Hour  of  Instr  Mat  Dev 

CIMD 

5 

F5.2 

143-144 

Pet  Instr  Material  Remaining  Value 

EVIM 

2 

F2.2 

145-149 

Supplies  Cost 

CSUPP 

5 

F5.2 

150-154 

Miscellaneous  Cost 

CM  IS 

5 

F5.2 

155-157 

Discount  Rate 

DRATE 

3 

F3.3 

158-160 

Inflation  Rate 

INRATE 

3 

F 3. 3 

161-168 

Facility  Init/Refurb  Cost 

FACST 

8 

F8.2 

169-288 

Graduates 

GRAD(I) 

120 

2016 

289-388 

0&MN  Annual  Maint  Costs 

COPMT(I) 

100 

20F5.0 

Record  Length  = 388  Positions 


FIGURE  1 1 1-6.  EXC  DATA  ELEMENTS 
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REC  POS 

VARIABLE  DESCRIPTION 

SYMBOL 

LENGTH 

FORMAT 

1-13 

Vehicle  Federal  Stock  Number 

VFSN 

13 

A13 

14-15 

Planning  Period  (Years) 

N 

2 

12 

16 

Equipment  Depreciation  Policy 

DEPF 

1 

11 

17-18 

Life  of  Equipment 

LOFEQ 

2 

12 

19-26 

Equipment  Unit  Cost 

EQUNTC 

8 

F8.2 

27-34 

Equipment  Installation  Cost 

EQINSC 

8 

F8.2 

35-37 

Discount  Rate 

DRATE 

3 

F3.3 

38-40 

Inflation  Rate 

INRATE 

3 

F3.3 

41-160 

Equipment  Planned  Purchases 

PEQUIP(I) 

120 

2016 

161-260 

0&MN  Annual  Maint  Costs 

COPMT(I) 

100 

20F5.0 

Record  Length  - 260  Positions 


i 


I 


FIGURE  1 1 1-7.  EXV  DATA  ELEMENTS 
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REC  POS 

VARIABLE  DESCRIPTION 

SYMBOL 

LENGTH 

FORMAT 

1-13 

Job/Task  Identifier 

JTID 

13 

,A13 

14-15 

Planning  Period 

N 

2 

12 

16-20 

Task  Performance  Time  (HRS) 

TTASK 

5 

F5.1 

21-25 

Task  Frequency/Person/Year 

FTASK 

5 

F5.0 

26-27 

Pet  Officers  Performing 

OPTSK 

2 

F2.2 

28-29 

Pet  Enlisted  Performing 

EPTSK 

2 

F2.2 

30-31 

Pet  Civilian  Performing 

CPTSK 

2 

F2.2 

32-34 

Error  Rate/Freq.  Performed 

ERTSK 

3 

F3.3 

35-42 

Non  Manpower  Cost/Error 

OERRC 

8 

F8.2 

43-49 

Officer  Billet  Cost 

OFFSL 

7 

F7.2 

50-56 

Enlisted  Billet  Cost 

ENLSL 

7 

F7.2 

57-63 

Civilian  Cost 

CIVSL 

7 

F7.2 

64-71 

Support  Cost/Year/Person 

Performing 

SUPPC 

8 

F8.2 

72-74 

Discount  Rate 

DRATE 

3 

F3.3 

75-77 

Inflation  Rate 

INRATE 

3 

F3.3 

78-197 

Annual  Personnel  Available 

APERS(I) 

120 

2016 

Record  Length  = 197  Positions 


FIGURE  1 1 1-8.  EXJ  DATA  ELEMENTS 
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PAGE  NUMBER  : 1 

SOURCE  CARO  LISTING  FOR 
**BPT 


**BPT 

REFERENCE  "EMPIRICAL  DATA"  SECTION  OF  ID  FILE  FOR  EXPERIMENTAL  RESULTS  AND 
INITIAL  PATTERN  OF  BENEFITS. 


BENEFIT  PATTERN 
TRAINING  BENEFITS 

I.  ATTRITION  RATE  - ESTIMATED  REDUCTION  APPROXIMATELY  80t  (FROM  7t  TO  It). 

THE  DOLLARS  SAVED  BY  THE  ATTRITION  REDUCTION  HAVE  BEEN 
ASSIGNEO  AN  IMPORTANCE  RATING  OF  10  OUE  TO  PRESENT 
FUNDING  PRESSURE. 

CONFIDENCE  RANGE  — BEST  CASE.  85t  REDUCTION— 

— WORST  CASE  50t  REDUCTION— 


2. '*  APfTnoDE  RQMTS  - 

3.  TRAINING  PROGRAM 
JOB  BENEFITS 

1.  ON-JOB-TRNG 


2.  STANDBY  JNITS 


LIABILITY  TO  REOUCTION  IN  ATTRITION  RATE:  INAPPROPRIATE 
EMPHASIS  ON  EVALUATING  STUDENT  ON  "THEORY  OF  OPERATION" 
CONTENT  IN  COURSE.  WOULD  NOT  AFFECT  ATTRITION  RATE  ON 
TARGETED  OBJECTIVE,  BUT  ON  THE  COURSE  ITSELF. 

NOTE:  IF  THE  REDUCTION  IN  ATTRITION  HAD  VALUE  OTHER 

THAN  IN  OOLLARS  SAVED,  THEN  IT  WOULD  HAVE  BEEN 
ASSIGNED  AN  IMPORTANCE  LEVEL  AS  OTHER  NONQUANT  I - 
F I ABLE  VARIABLES  ARE.  THIS  MIGHT  BE  THE  CASE 
WHERE  THE  PARTICULAR  MANPOWER  POOL  IS  IN  SHORT 
SUPPLY. 

THERE  IS  A POTENTIAL  FOR  REDUCTION  BUT  IT  IS  NOT 
QUANTIFIABLE  AT  THIS  TIME. 

DEVELOPMENT  FLEXIBILITY  - INCREASED  - IMPORTANCES. 


ESTIMATE  NORMAL  OJT  BEFORE  ACCEPTABLE  PERF  - 4 WEEKS 
ESTIMATE  OF  OJT  REQUIRED  WITH  INNOVATION  - 1 WEEK 

THE  DOLLARS  SAVED  BY  THE  OJT  REDUCTION,  AS  WITH  ATTRI- 
TION OOLLARS,  HAVE  BEEN  ASSIGNED  AN  IMPORTANCE  OF  10. 

CONFIDENCE  RANGE  — BEST  CASE,  .5  WEEK  WITH  INNOVATION- 
— WORST  CASE,  1.5  WEEKS  — 

REDUCED  - IMPORTANCES. 

CONFIDENCE  RANGE  — BEST  CASE,  50t  REDUCTION— 

--  WORST  CASE,  lOt  REDUCTION- 


FIGURE  111-9.  BPT  FILE  EXAMPLE 
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The 


file 

is  forma 

Pos 

1 

Pos 

2-  14 

Pos 

15-  19 

Pos 

20-  21 

Pos 

22-201 

Pos 

202-381 

?d  as  follows: 

Cost  Model  Type  Output;  i.e.,  Training  (Courses)(l ) , 
Vehicle(2),  Job/Task(3),  Risk  Projects(4),  Course  Total (5), 
Vehicle  Total(6),  Job/Task  Total(7),  Entity  Total  ( 5+6+7 ) (8 ) , 
Project  Total  (4+8)(9). 

Entity  Identifier;  e.g.  CDP,  FSN,  "Course  Total",  etc. 

Count  of  Entities  Summarized 

Planning  Period  in  Years  (Indicate  Longest  in  Summaries). 
Investment  Vector  (20-9  Pos  $ Fields  for  20  Years) 

Savings  Vector  (20-9  Pos  $ Fields  for  20  Years) 


RKP  (Risk  Profile).  The  introduction  of  an  innovation  will  have  attendant 
risks  which  affect  the  success  probabilities  associated  with  its  implemen- 
tation and  acceptance.  The  nature  and  degree  of  risk  will  assist  in  struc- 
turing risk  reduction  projects.  A summary  of  the  degree  of  risk  by  risk 
category,  the  number  of  risk  projects  which  have  been  identified  for  each 
risk  category,  and  the  estimated  cost  of  the  projects  in  each  risk  category 
appear  at  the  beginning  of  the  RKP  file.  There  are  seven  identified  risk 
categories  numbered  1 through  7 in  the  summary.  A number  0 card  contains 
header  information  and  a number  8 card  gives  an  overall  profile  of  the  risk 
As  many  number  9 cards  as  desired  can  be  used  to  provide  descriptive  infor- 
mation on  the  nature  of  the  risks  in  each  category.  An  example  of  the  con- 
tents of  the  BPT  file  is  shown  in  Figure  1 1 1 - 1 0 . This  file  can  be  printed 
with  the  PI  print  program. 


RRPJ  (Risk  Reduction  Projects).  This  card-image  file  contains  descriptive 
and  resource  data  associated  with  each  risk  reduction  project  proposed.  Type 
0 records  in  this  file  contain  the  project  identification,  project  name, 
date,  and  dollar  value  of  the  project.  Type  1 records  will  contain  descrip- 
tive data  on  the  project  including  an  abstract,  objectives,  project  plan 
milestones,  etc.  Type  2 through  7 records  in  the  RRPJ  file  will  contain 
specific  resource  requirements  by  resource  category.  Type  2 records  are  for 
personnel  resources.  Type  3 records  are  for  equipment  resources.  Type  4 
records  are  for  space  resources.  Type  5 records  are  for  staff  travel  resources, 
Type  6 records  are  for  student  resources,  and  Type  7 records  are  for  other 
resources . 


The  file  is  formatted  as  follows: 


All  Record  Types- 


Pos 

1 

Record  Type  (0-7) 

Pos 

2 

- 

Type  0 Record- 

Pos 

3-  10 

Project  Identification 

Pos 

11-  12 

Blank 

Pos 

13-  58 

Project  Name 

Pos 

59-  60 

Blank 

Pos 

61-  68 

Date  (MM/DD/YY) 

Pos 

69-  71 

Blank 

Pos 

72-  80 

Cost  of  Project 
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**RKP 


• RKP 

-RISK  CATEGORY  / DEGREE  OF  RISK  - h: 

-IMPORTANCE  TO  PROJECTED  NAVY  MISSION 

-ORGANIZATIONAL  COMPATIBILITY 

-GOALS/POLICY  COMPATIBILITY 

-STATE-OF-THE-ART 

-RCD  FUNDING  REQUIREMENTS 

-TECHNICAL  SUPPORT  REQUIREMENTS 

-ATT l TUDI NAL  ACCEPTANCE 

-OVERALL  PROFILE-TOTAL  PROJECTS/DOLLARS 

-FOLLOWING  IS  DESCRIPTIVE  INFORMATION  ON 


[GH 

MOD 

NONE 

SUPPOR 

T PRO  J 

COST 

0 

I 

0 

0 

0 

0 

1 

0 

00 

000000000 

0 

0 

0 

I 

00 

000000000 

0 

1 

0 

0 

04 

11000000 

0 

1 

0 

0 

00 

000000000 

0 

1 

0 

0 

00 

000000000 

0 

1 

0 

0 

01 

55000 

0 

I 

0 

0 

05 

11055000 

EACH 

RISK 

AREA 

SUMMAR 

I ZED  ABOVE. 

IMPORTANCE  TO 
OF  MODERATE 


PROJECTED  NAVY 
IMPORTANCE 


MISSION 


ORGANIZATIONAL  COMPATIBILITY 

IRRELEVANT  TO  NAVY  COMMAND  AND  CAREER  STRUCTURES 

GOALS/POLICY  COMPATIBILITY 

SUPPORT  OF  POLICY  TO  INCREASE  PRACTICAL  SKILL  LEVELS 
STATE-OF-THE-ART 

MODERATE  PROBABILITY  OF  SUCCESS  WITH  SOME  ADDITIONAL  RED  EFFORT  REQUIRED 
PROBLEMS: 

HARDWARE  LINKAGES  OF  EQUIPMENT  TO  COMPUTER  SENSORS 

TRAINING  LABORATORY  LOCATIONS  WILL  PRESENT  SOME  PROBLEMS  IN  TYING 
STUDENT  STATIONS  TO  A REMOTE  COMPUTER  FACILITY.  PRESENT  TECHNOLOGY 
PERMITS  LINE  LENGTHS  OF  UP  TO  2000  FEET  TO  BE  DRIVEN  AT  REASONABLE  COST. 
SOME  FACILITIES  ARE  A HALF-MILE  FROM  THE  COMPUTER  SITE.  RCD  EFFORTS 
SHOULD  BE  DEVOTEO  TO  FINDING  A TECHNOLOGY  CAPABLE  OF  DRIVING  LINES  UP  TO 
ONE  MILE  AT  REASONABLE  COST. 

SIMULATION  OF  TEMPORAL  RESPONSE  TO  OPERATION  ACTIONS 

PRESENT  COMPUTATIONAL  SPEEDS  WOULD  CAUSE  DELAYS  OF  UP  TO 
RESPONDING  TO  OPERATION  ACTIONS.  THIS  MAY  AFFECT  STUDENT 
THE  DEVICE,  BUT  MORE  SO,  MAY  IMPAIR  THE  LEARNING  OF  CERTAI 
PROCEDURES  WHERE  A SEQUENCE  OF  ACTIONS  MUST  BE  LEARNED  AND 
RAPID  FEEDBACK  MAY  BE  ESSENTIAL.  AN  ASSESSMENT  (STUDY)  OF 
MATE  AMOUNT  OF  TRAINING  TAKING  PLACE  ON  ACTUAL  EQUIPMENT, 

RESPONSE  THAN  IS  NOW  POSSIBLE,  SHOULD  BE  MADE.  ADDITIONAL 
ALSO  BE  DIRECTED  AT  DETERMINING  STUDENT  ATTITUDE  EFFECTS, 

THEY  MIGHT  RELATE  TO  DECREASED  LEARNING  EFFECTIVENESS. 

RELIABILITY  OF  H AR DWAR E / SOF T W ARE  PERFORMANCE 

AVAILABILITY  OF  STUDENT  TRAINERS  WILL  BE  THE  KEY  TO  ACHIEVING  DESIRED 
OBJECTIVES.  PRESENT  COMPUTER  RELIABILITY  WOULD  APPEAR  TO  INDICATE  AN 
AVAILABILITY  OF  BETWEEN  90%  AND  95%.  IT  MUST  BE  DETERMINED  IF  SUFFI- 
CIENT TRAINING  CONTINGENCY  PLANS  CAN  BE  ADOPTED  TO  COPE  WITH  THIS  LEVEL 
OF  RELIABILITY. 

LEAD  TIME  NEEDED  TO  PROGRAM  THE  TRAINING  FUNCTIONS  ON  DEVICE 

THE  PROPOSED  TRAINER  IS  GENERAL  PURPOSE  AND  ITS  UNIQUENESS  TO  THE 
TRAINING  SITUATION  WILL  PRIMARILY  RESULT  FROM  PROGRAMMING  THE  INTER- 
RELATIONSHIPS BETWEEN  OPERATOR  CONTROLS  AND  FEEDBACK  SIGNALS  (LlC-h.'S, 
METERS,  ETC.).  PROGRAMMING  TIME  WAS  ASSESSED  FOR  THE  EXPERIMENTAL 
APPLICATION,  BUT  NO  SPECIFIC  TECHNIQUES  WERE  ANALYZED  FOR  SPEEDING  THE 
PREPARATION  OF  PROGRAMS.  A MAJOR  STUDY  SHOULD  BE  UNDERTAKEN  TO  INVESTI- 
GATE POSSIBLE  TECHNIQUES  FOR  IMPROVING  PROGRAMMING  EFFICIENCY. 

FUNDING 

LARGE  OR  VERY  LARGE  INVESTMENT  TO  BRING  THE  INNOVATION  TO  AN  OPERATIONAL 
LEVEL  OF  i MPLEMENT  AT  ION  (NEAR  MAXIMUM  THAT  CAN  BE  COMMITTED  TO  ONE  PROJECT). 

TECHNICAL  SUPPORT  REQUIREMENTS 

SOME  DIFFICULTY  IN  GETTING  RED  TALENT  AND  IMPLEMENTATION  TALENT. 


4 SECONDS  IN 
ACCEPTANCE  OF 
N OPERATIONAL 
WHERE  MORE 
THE  APPROX  I - 
WHICH  FASTER 
STUDY  SHOULD 
ESPECIALLY  AS 


FIGURE  111-10.  RKP  FILE  EXAMPLE 
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SOURCE  CARD  LISTING  FOR 
**RKP 

- ATTITUDINAL  ACCEPTANCE 

MODERATE  BUT  TEMPORARY  RESISTANCE  BY  INSTRUCTIONAL  COMMUNITY,  ESPECIALLY 

- IF  THERE  WILL  BE  DIFFICULTIES  IN  OPERATION,  EQUIPMENT  RELIABILITY,  OR  SYSTEM 

- AVAILABILITY  AND  RESPONSIVENESS. 

PROBLEMS: 

STUDENT  ACCEPTANCE 

ACCEPTANCE  OF  THE  INSTRUCTIONAL  COMMUNITY  WILL,  TO  A GREAT  EXTENT,  BE 
A FUNCTION  OF  SUCCESSFULLY  OVERCOMING  THE  RISK  ITEMS  STATEO  ABOVE. 
EXCESSIVE  SYSTEM  DOWNTIME  OR  SLOW  RESPONSE  TIME  COULD  SERIOUSLY  IMPACT 
STUDENT  ACCEPTANCE.  APPROPRIATE  ORIENTING  PRESENTATIONS  SHOULD  BE  PRE- 
PARED TO  HELP  OVERCOME  RESISTANCE  OF  ADMINISTRATORS  AND  STUDENTS. 


FIGURE  111-10.  RKP  FILE  EXAMPLE  (Cont'd) 
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Type  1 Record- 

Pos  3-  80  Descriptive  Data 
Type  2-7  Record- 


Pos 

•3-  47 

Resource  Descriptive  Data 

Pos 

48-  49 

Blank 

Pos 

50-  58 

Resource  Quantity  Data 

Pos 

59-  60 

Blank 

Pos 

61-  69 

Unit  Cost 

Pos 

70-  71 

Blank 

Pos 

72-  80 

Job  Cost 

An  example  of  the  contents  of  the  RRPJ  file  is  shown  in  Figure  1 1 1 - 1 1 . 

RRPK  (Risk  Reduction  Packages).  The  projects  defined  in  the  RRPJ  file  are 
grouped  into  various  packages,  each  of  which  will  have  some  level  of  effect 
upon  the  implementation  and  user  acceptance  success  probabilities  which  become 
a part  of  the  decision  tree  evaluation.  This  file  also  contains  the  probability 
factors  associated  with  (1)  risk  package  implementation  success,  (2)  effect 
upon  innovation  implementation  success,  and  (3)  effect  upon  success  of  the 
innovation  considering  its  acceptance  by  the  using  system.  The  file  is 
formatted  as  follows: 


Pos 

4 

Pos 

5- 

10 

Pos 

11- 

12 

Pos 

13- 

20 

Pos 

21- 

80 

3 

1 

Pos 

4- 

12 

Pos 

13- 

20 

Pos 

21- 

22 

Pos 

23- 

30 

Pos 

31- 

32 

Pos 

33- 

40 

Pos 

41- 

42 

Pos 

43- 

51 

Risk  Reduction  Package  Number  (0-9) 

0 for  1st  Card  of  Any  Risk  Reduction  Package 

Blank 

10  Risk  Package  Designator 

12  Blank 

20  Project  Identifier  (A  Position  1 = 0 Record 
has  special  header  info  - See  Figure  1 1 1 - 1 2 ) 

80  Repeat  of  Pos  11-20 

1 for  2nd  Card  of  Any  Risk  Reduction  Package 

12  Blank 

20  PPp  PPm  PP  three  probabilities  related 

to  the  expected  success  of  the  Risk  Reduction 
Package.  PPp  is  the  pessimistic  probability, 
PPm  the  expected,  and  PP0  the  optimistic. 

22  Blank 

30  Three  probabilities  as  in  Pos  13-20  related 

to  the  expected  success  of  being  able  to 
implement  the  innovation  given  that  the 
identified  Risk  Reduction  Package  succeeds. 

32  Blank 

40  Three  probabilities  as  in  Pos  13-20  related 

to  the  expected  success  of  gaining  user  accep- 
tance of  the  innovation,  once  the  innovation 
is  implemented. 

42  Blank 

51  Planned  cost  of  All  Risk  Reduction  Projects  in 

the  particular  package. 
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SOURCE  CARO  LISTING  FOR 
**RRPJ 


**RRPJ 

0- OOI-ATTD  USER  ORIENTATION  FOR  3-D  TRAINER  5/02/77  55000 

1- 

1-ABSTRACT 

I-  IMPLEMENTATION  OF  THE  3-0  PROCEDURAL  TRAINER  THROUGHOUT  THE  TRAINING 
l-COMMAND  WILL  BE  ACCOMPANIED  BY  A GREAT  DEAL  OF  RESISTANCE,  ESPECIALLY  AT  THE 
I -COMMAND  LEVELS.  ONE  SOLUTION  TO  THIS  PROBLEM  IS  TO  ENSURE  THAT  AN  ADEQUATE 
1 -UNDERSTANDING  OF  THE  OEVICE  EXISTS  WITH  OFFICERS  AND  STAFF  WHO  WILL  BE  RESPON- 
SIBLE TO  MANAGE  THE  IMPLEMENTATION  AND  OPERATION  OF  THESE  DEVICES  WITHIN  THEIR 
1-COMMAND.  THIS  WILL  BE  ACCOMPLISHED  BY  DEVELOPING  A 15-MINUTE  COLOR  MOVIE  ON 
l-THE  DEVICE,  ITS  FEATURES,  AND  ITS  APPLICATION,  FOR  PRESENTATION  TO  COMMAND 
1-LEVEL  OFFICERS  AND  STAFF  PERSONNEL  PRIOR  TO  THE  PLANNING  STAGE  FOR  ONE  OF 
1-THESE  DEVICES.  THREE  PERSONS  WILL  REQUIRE  A TOTAL  OF  12-MAN  MONTHS  TO  DEVELOP 
l-THE  MOVIE  AT  AN  ESTIMATED  C3ST  OF  S55.000.  DEVELOPMENT  WILL 
1-BEGIN  MAY  1977  AND  COMPLETE  APRIL  1978. 
l-OBJECTIVES 

1-  DEVELOP  15-MINUTE  COLOR  MOVIE  ON  3-D  PROCEDURAL  TRAINER. 

1-  IMPROVE  ACCEPTANCE  OF  USERS  AT  COMMAND  LEVEL. 

l-PROJECT  PLAN  MILESTONES 

1-  PREPARE  FINAL  IMPLEMENTATION  PLAN.  JAN 

1-  OBTAIN  FUNDING  APPROVAL  MAR 

1-  RECRUIT  PERSONNEL.  APR 

1-  BEGIN  OESIGN -CONSULTATION  TRIPS.  MAY 

1-  COMPLETE  DESIGN.  OCT 

l-  BEGIN  PHOTOGRAPHY.  NOV 

1-  COMPLETE  PHOTOGRAPHY.  FEB 

1-  OBTAIN  PRINTS.  APR 

1-  DISTRIBUTE  PRINTS  FOR  USE.  MAY 

1- 

2- PERSONNEL  -JOB- 

2-  MOVIE  DESIGN  LAYOUT  (GS-12) 

2-  PHOTOGRAPHY  ( PH6 J 

2-  ADMINISTRATIVE 
2- 

3- EQUI PMENT  -DESCRIPTION- 
S' PROTOTYPE  DEVICE  FOR  USE  IN  OESIGN 

3-  OF  SEQUENCES  AND  FILMING 

3- 

A-SPACE  -REASON- 

A- 

5-TRAVEL-STAFF  -FR0M-T0-REAS3N 
5-  MULTI-LOCATION  TRAVEL  FOR  CONSULTATION/FILM 

5- 

6- STUDENT  -NO.  OF  STUDENTS-REASON 
6- 

7- OTHER  -DESCRIPTION- 

7-  PROCESSING  AND  REPRODUCTION 

7-  OF  100  COPIES  OF  THE  FILM 
7- 

0- 00 1-SART  NEXT  PROJECT 

1- 


BE  SCHEDULED  TO 


1977 

1977 

1977 

1977 

1977 

1977 

1978 
1978 
1978 


IAN  MONTHS 

$/  MM 

JOB  COST 

6 

1855 

11130 

A 

1630 

6520 

2 

970 

1 9 AO 

QUANTITY 

UNIT  COST 

JOB  COST 

1 

12A00 

12A00 

SQ  FT 

S/ SQ  FT 

JOB  COST 

NO.  TRIPS 

S/TRIP 

JOB  COST 

20 

300 

6000 

NO.  DAYS 

S/DAY 

JOB  COST 

MM/QTY 

UNIT  COST 

JOB  COST 

1 

6010 

6010 

100 

no 

11000 

FIGURE  111-17.  RRPJ  FILE  EXAMPLE 
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An  example  of  the  contents  &f-the  RRPK  file  is  shown  in  Figure  111-12. 

VQAL  (Variables  Qualitative).  This  card-image  file  contains  descriptive  data 
on  the  qualitative  variables  impacted  by  the  adoption  of  the  proposed  innova- 
tion. These  data  will  be  reviewed  by  the  assessor  along  with  the  cost  model 
results  in  the  scaling  process.  In  sequence  at  the  beginning  of  this  file 
is  a list  which  may  have  potential  for  being  affected  by  implementing  the 
innovation.  The  format  for  this  part  of  the  file  is: 

Pos  1-  3 Variable  Reference  Number  001-998 

000  = COURSES  001-399  are  reserved  for  training 
related  variables.  400  = VEHICLES  401-699  are 
reserved  for  vehicle  related  variables.  700  = 
JOB/TASKS  701-998  are  reserved  for  job/ task 
related  variables.  999  indicates  the  end  of 
the  variable  list. 

Pos  4 Blank 


Pos  5-  33  Variable  Name  - Names  should  be  limited  to  29 

characters  to  fit  in  name  field  of  VARF  file. 

The  remainder  of  the  file  contains  the  specific  qualitative  variables  associated 
with  the  project  which  are  planned  for  incorporation  into  the  Decision  Tree 
utility  outcomes. 


The  format  for  this  part  of  the  file  is: 


Pos  1-  3 Variable  Reference  Number  - 001-998 

Pos  4 Blank 


Pos  5-  80  Descriptive  Data  - The  first  of  possible  multiple 

records  will  contain  the  Variable  Name  in  Pos  5-33. 
Each  record  related  to  the  same  variable  will  contain 
the  same  Variable  Reference  Number  in  Pos  1-3. 


An  example  of  the  contents  of  the  VQAL  file  is  shown  in  Figure  1 1 1 - 1 3 . 

VARF  (Variable  Reference).  Each  of  the  variables,  including  costs  will  be 
valued  during  the  interactive  scaling  process.  An  importance  weighting  factor 
and  an  equivalent  dollar  (utility)  value  will  be  developed  for  each  of  the  four 
possible  decision  tree  outcomes.  The  summation  of  the  outcome  values  stored  in 
this  file  for  each  variable  will  be  used  in  the  decision  tree  evaluation. 

The  format  of  this  card-image  file  is  as  follows: 

Pos  1-  3 Variable  Reference  Number  - a 000  in  these  positions 

refers  to  the  direct  cost  results  from  1)  running 
the  cost  models  and  identifying  the  savings  if  the 
full  benefit  pattern  is  achieved,  and  2)  interactive- 
ly (or  in  batch)  assigning  cost/savings  which  occur 
at  the  Decision  Tree  outcomes  where  the  full  benefit 
pattern  (and  most  likely  several  liabilities)  does 
not  occur. 
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SOURCE  CARO  LISTING  FOR 
♦ORRPK 


♦*RRPK 

0-0  NORISK  PROJ-SUC  IMPL-SUC  USER-SUC  PROJ-COST 

0- 1  00  00  00  65  75  85  40  50  60 

1 - 0  RKPKGI  OOI-SART  002-SART  003-SART  004-SART  OOI-ATTO 

1- l  90  95  99  80  85  90  50  60  70  11055000 

2- 0  RKPKG2  OOI-SART  003-SART  004-SART 

2- 1  90  95  99  80  85  93  50  60  70  9000000 

3- 0  RKPKG3  002-SART  003-SART  OOI-ATTO 

3- 1  90  95  99  70  75  80  80  85  95  3055000 

4- 0  RKPKG4  PROJECT  GROUP  4 IF  APPLICABLE 

4- 1  PROJ-SUC.  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  4 

5- 0  RKPKG5  PROJECT  GROUP  5 IF  APPLICABLE 

5- 1  PROJ-SUC,  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  5 

6- 0  RKPKG6  PROJECT  GROUP  6 IF  APPLICABLE 

6- 1  PROJ-SUC,  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  6 

7- 0  RKPKG7  PROJECT  GROUP  7 IF  APPLICABLE 

7- 1  PROJ-SUC,  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  7 

8- 0  RKPKG8  PROJECT  GROUP  8 IF  APPLICABLE 

8- 1  PROJ-SUC,  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  8 

9- 0  RKPKG9  PROJECT  GROUP  9 IF  APPLICABLE 

9-1  PROJ-SUC,  IMPL-SUC,  USER-SUC,  PROJ-COST  IF  APPLICABLE  FOR  PROJ  GRP  9 


FIGURE  111-12.  RRPK  FILE  EXAMPLE 
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PAGE  NUMBER  : 1 

SOURCE  CARD  LISTING  FOR 

**VQAL 


**VQAL 

000  COURSES 

AVAIL  OF  TRND  PERSONNEL 
FLEX  IN  P ERF  TRNG  OPNS 
SKILL  LEVEL  OF  INSTR  PERS 
TRAINING  QUALITY 
STUDENT  MORALE 
MOTIVATION 

STUDENT  LRNG  RETENTION 
INSTR  APTITUDE  RQMTS 
TRAINING  SAFETY 
COURSE  SCHEDULING 
ACCEPTANCE  OF  TRAINING 
VEHICLES 

PORTABILITY  OF  EQUIP 
PROGRAMMABIL ITY 
FLEXIBILITY  IN  USAGE 
MAINTAINABILITY 
UPDATE  TO  INSTR  FUNCTION 
TRNG  CONTENT  MODULARIZATN 
JOB/TASKS 

FLEX  IN  WORK  ASSIGNMENT 
RETENTION  OF  SKILL 
AVAIL  IN  STANDBY  EQUIP 
JOB  STRESS 
END 

FLEXIBILITY  IN  PERFORMING  OTHER  TRAINING 
THE  3-D  PROCEDURAL  TRAINER  WILL  PERMIT 
DEVELOPED  MODULARLY,  ANO  MOOULES  WILL  BE 

ALSO,  HIGHER  LEVEL  PROBLEM  PREPARATION  LANGUAGES  WILL  FURTHER  INCREASE 
THE  FLEXIBILITY  IN  TRAINING  PROGRAM  DEVELOPMENT. 


001 

002 

003 

00* 

005 

006 

007 

008 

009 

010 
Oil 
*00 
*01 
*02 
*03 
*0* 
*05 
*06 

700 

701 

702 

703 
70* 
999 
002 
002 
002 
002 
002 
002 
703 
703 
703 


RELATED  OPERATIONS 
INSTRUCTION  PROGRAMS  TO  BE 
USABLE  ACROSS  TRAINING  COURSES. 


INCREASED  SKILLS  DEVELOPED  THROUGH  IMPLEMENTATION  OF  THE  PROCEDURAL 
TRAINER  SHOULD  IMPROVE  EQUIPMENT  TO  BE  REDUCED  BY  APPROXIMATELY  30f. 


FIGURE  111-13.  VQAL  FILE  EXAMPLE 
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A number  from  001  through  998  references  a qualitative 
variable  stored  in  the  VQAL  project  file.  The  number 
999  is  reserved  for  a total  of  the  VARF  cost  and  equi- 
valent cost  data.  This  record  will  be  an  input  to 
the  Decision  Analysis  routine's  final  decision  tree 
calculation. 


Pos 

4 

Blank 

Pos 

5-  6 

Importance  Weighting  - this  is  a two  digit  number  from 
1 through  10  which  will  be  used  to  weight  the  equivalent 
dollars  (utility)  associated  with  the  variable.  It  is 
inputted  via  card  or  developed  interactively. 

Pos 

7 

Blank 

Pos 

8-  16 

Decision  Tree  Outcome  A Value  - this  is  the  value  that 
this  particular  variable  takes  on  after  it  has  been 
evaluated  in  relation  to  actual  costs,  and  then 
weighted  by  its  importance. 

Pos 

17 

Blank 

Pos 

18-  26 

Decision  Tree  Outcome  B Value  - same  as  Pos  8-16 
except  it  represents  the  weighted  equivalent  dollars 
for  Outcome  B of  the  decision  tree. 

Pos 

27 

Blank 

Pos 

28-  36 

Decision  Tree  Outcome  C Value  - same  as  Pos  8-16 
except  it  represents  the  weighted  equivalent  dollars 
for  Outcome  C of  the  decision  tree. 

Pos 

37 

Blank 

Pos 

38-  46 

Decision  Tree  Outcome  D - always  zero,  since  this 
is  the  Reject  outcome  on  the  decision  tree. 

Pos 

47-  51 

Blank 

Pos 

52-  80 

Variable  Name  - the  short  name  for  the  variable 

(001  = COST)  from  the  VQAL  project  file.  A 999 

in  Pos  1-3  will  have  the  term  TOTAL  in  these  positions. 


An  example  of  the  contents  of  the  VARF  file  is  shown  in  Figure  1 1 1 - 1 4 . 

SCEN  (Scenarios).  This  is  a file  in  which  descriptions  of  each  of  the  scenarios 
developed  for  tne  decision  tree  outcomes  are  stored.  Up  to  four  scenarios  can 
be  created,  although  generally  the  Reject  decision  path  assumes  a no  change 
situation.  The  contents  of  this  file  will  be  used  by  the  assessor  in  creating 
the  values  for  each  of  the  decision  tree  outcomes.  The  data  are  stored  in 
80-column  card  image  format  with  major  subdivisions  made  for  each  outcome  (A,  B, 
C and  D).  An  example  of  the  SCEN  file  is  shown  in  Figure  III-15. 
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PAGE  NUMBER  : 1 

SOURCE  CARD  LISTING  FOR 

**V ARF 


**VARF 


000 

10 

46669728 

1753862 

-1000000 

00 

COST 

002 

02 

8500000 

-400000 

-1000000 

00 

FLEX  IN  PERF  TRNG  OPNS 

703 

01 

3800000 

-1000000 

-500000 

00 

AVAIL  IN  STANDBY  EQUIP 

999 

58969723 

353862 

-2500000 

00 

TOTAL 

FIGURE  111-14.  VARF  FILE  EXAMPLE 
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PAGE  NUMBER  : 1 

SOURCE  CARD  LISTING  FOR 
**SCEN 


**SCEN 

THE  FOLLOWING  SCENARIOS  REPRESENT  THE  QUANTITATIVE  AND  QUALITATIVE  EXPECTED 
OUTCOMES  AT  EACH  LEG  OF  THE  DECISION  TREE. 


A - ATTRITION  WILL  BE  REDUCED  FROM  .07  TO  .01. 

INSTR. /STUD.  POSITION  RATIO  WILL  GO  FROM  .25  TO  .05. 

EQUIPMENT  LIFE  WILL  GO  FROM  5 YEARS  TO  10  YEARS. 

EQUIPMENT  UNIT  COST  WILL  GO  FROM  44375.  TO  45625. 

OJT  TIME  WOULO  BE  REOUCEO  FROM  4 WEEKS  TO  1 WEEK. 

ALL  OTHER  COST  DATA  FROM  THE  FORMERLY  PLANNED  PROGRAM  TO  THE  PROPOSED 
PROGRAM  WILL  REMAIN  THE  SAME. 

STANDBY  EQUIPMENT  - THIS  QUAL I T I AT  I VE  VARIABLE  WHICH  MEASURES  THE  GREATER 
AVAILABILITY  OF  STANDBY  EQUIPMENT  IN  THE  FLEET  WILL  BE  IMPROVED  OVER  THE 
PRESENT  STATE  BY  ABOUT  5 TO  1,  ALTHOUGH  IT  IS  ONLY  ABOUT  1/10-TH  AS 
IMPORTANT  AS  THE  ATTRITION  AND  OJT  TIME  SAVINGS  WHICH  ARE  EXPECTED  TO 
ACCRUE. 

FLEXIBILITY  IN  TRAINING  PROGRAM  DEVELOPMENT  - THIS  QUAL I T I TAT  I VE  VARIABLE 
WHICH  MEASURES  THE  GREATER  FLEXIBILITY  THAT  PROGRAM  DEVELOPERS  WILL  HAVE  IN 
PREPARING  COURSE  CURRICULUM  WILL  BE  IMPROVED  OVER  THE  PRESENT  STATE  BY  ABOUT 
10  TO  l,  ALTHOUGH  ITS  IMPORTANCE  IS  ONLY  ABOUT  1/5-TH  THAT  OF  THE  ATTRITION 
AND  OJT  TIME  SAVINGS  WHICH  ARE  EXPECTED  TO  ACCRUE. 


B - ATTRITION  COULO  FAIL  TO  REACH  .01  AND  WILL  STAY  AROUND  .04. 

TWO  YEARS  WILL  BE  REQUIRED  TO  DETERMINE  ACCEPTANCE. 

RECOVERY  WILL  BE  ACCOMPLISHED  BY  RE  I NST I TUT  I NG  THE  BASELINE  OR  EXISTING  PLAN 
ON  WHICH  THE  IMPROVEMENTS  IN  OUTCOME  A WERE  BASED. 

TRAINERS  WILL  HAVE  NO  SALVAGE  VALUE  AT  END  OF  TWO-YEAR  PERIOD. 

NO  OJT  TIME  WILL  BE  SAVED. 

STANDBY  EQUIPMENT  - IF  THE  PROGRAM  IS  INSTITUTED  AND  FAILS  TO  GAIN  USER 
ACCEPTANCE,  THERE  WILL  BE  A DECREASED  AVAILABILITY  OF  EQUIPMENT  FOR  A 
CONSIDERABLE  PERIOD  OF  TIME.  THIS  WILL  PRODUCE  A STATE  WHICH  IS  ABOUT 
20*  AS  NEGATIVE  AS  OUTCOME  A IS  POSITIVE. 

FLEXIBILITY  IN  TRAINING  PROGRAM  DEVELOPMENT  - THIS  WILL  BE  AFFECTED  IN  A 
NEGATIVE  MANNER  ABOUT  3*  TO  5X  OF  THE  POSITIVE  VALUE  THAT  WILL  BE  ACHIEVED 
BY  OUTCOME  A. 


C - APPROXIMATELY  4700,000  IN  DEVELOPMENT  COSTS  WILL  BE  INCURRED. 

APPROXIMATELY  4300,000  WILL  BE  REQUIREO  TO  EXPEDITE  EQUIPMENT  PROCUREMENT 
TO  REVERT  TO  EXISTING  PROGRAM  PLAN. 

NO  OJT  TIME  WILL  BE  SAVED. 

STANDBY  EQUIPMENT  - THIS  OUTCOME  WILL  PRODUCE  A NEGATIVE  EFFECT  ABOUT  HALF 
OF  THAT  OF  OUTCOME  B. 

FLEXIBILITY  IN  TRAINING  PROGRAM  DEVELOPMENT  - THIS  OUTCOME  WILL  BE  2 TO  3 
TIMES  AS  NEGATIVE  AS  OUTCOME  0. 


0 - NEUTRAL  OUTCOME.  PRESENT  STATE  CONTINUES  TO  EXIST. 


FIGURE  111-15.  SCEN  FILE  EXAMPLE 
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TRER  (Tree  Results).  The  decision  variables  calculated  during  the  process  of 
folding  back  the  decision  tree  are  stored  in  this  file.  The  variables  to  be 
stored  in  this  file  are  discussed  later  in  the  Decision  Analysis  section.  They 
are  a summary  of  the  information  printed  on  the  Decision  Tree  Report  DT01  which 
is  an  output  of  Program  P60.  Each  record  contains  the  input  and  calculated 
values  associated  with  a single  decision  alternative;  i.e.,  the  alternative 
with  NO  RISK  PKG,  and  up  to  nine  RISK  PKG  alternatives.  Various  calculational 
runs  can  be  stored  and  are  given  a unique  sequence  number. 

The  format  of  this  file  is  as  follows: 


Pos 

1-  2 

Calculational  Sequence  Run  Number  - Up  to  100 
sequences  (00-99)  can  be  stored,  the  00  sequence 
will  be  the  baseline  run  from  the  data  stored  in 
the  CMR,  RRPK,  and  VARF  files. 

Pos 

3 

Risk  Package  Number  within  Sequence  Run  Number  - 
0 represents  the  NO  RISK  PKG  alternative.  1-9 
represent  the  nine  possible  RISK  PKG  conditions. 

Pos 

4 

Rank  of  Alternative 

Pos 

5-  13 

Utility  of  Outcome  A 

Pos 

14-  22 

Utility  of  Outcome  B 

Pos 

23-  26 

Probabilities  X and  Z 

Pos 

27-  35 

Value  X 

Pos 

3b-  44 

Value  Z 

Pos 

45-  53 

Utility  of  Outcome  C 

Pos 

54-  57 

Probabilities  U and  Y 

Pos 

58-  66 

Value  W 

Pos 

67-  75 

Value  Y 

Pos 

76-  84 

Utility  of  Outcome  D 

Pos 

85-  93 

Value  of  PSUCC  Path 

Pos 

94-  102 

Value  of  PFAIL  Path 

Pos 

103-  104 

Probability  of  Project  Success 

Pos 

105-  113 

Value  Before  Risk  Package 

Pos 

114-  122 

Cost  of  Risk  Package 

Pos 

123-  131 

Value  of  Decision  Variable 
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Pos  132-  133  Blank 

Pos  134-  255  Descriptive  Data  indicating  run  conditions.  Pos 

134-141  of  the  NO  RISK  PKG  alternative  (0  in  Col  3) 
for  each  Sequence  Run  Number  will  have  the  date  of 
the  run  stored  as  MM/DD/YY. 


ALTP  (Alternative  Projects).  This  file  stores  the  project  identifiers  for 
potential  alternatives  to  the  proposed  innovation.  Alternative  projects  will 
be  described  in  identical  fashion  to  the  primary  innovation.  The  contents 
of  this  file  will  identify  the  projects  which  may  be  compared  by  the  financial 
analysis  routine.  Up  to  five  alternative  project  numbers  may  be  stored. 

The  format  of  the  file  which  can  be  inputted  by  card  is  as  follows: 


Pos 

1-  5 

1st  Alternative 

Project 

ID; 

Pos 

6 

Blank 

Pos 

7-  11 

2nd  Alternative 

Project 

ID 

Pos 

12 

Blank 

Pos 

13-17 

3rd  A1 ternati ve 

Project 

ID 

Pos 

18 

Blank 

Pos 

19-  23 

4th  A1 ternati ve 

Project 

ID 

Pos 

24 

Blank 

Pos 

25-  29 

5th  Alternative  Project 

ID 

e.g.,  PN124 


An  example  of  the  contents  of  this  file  is  shown  in  Figure  1 1 1 - 1 6 . 

F I NR  (Financial  Results).  The  outputs  of  the  financial  analysis  routine  comparing 
various  alternative  innovation  proposals  are  stored  in  this  file.  The  variables 
to  be  stored  in  this  file  are  discussed  later  in  the  Financial  Analysis  section. 
They  are  a summary  of  the  information  printed  on  the  Financial  Analysis  Report 
FA01  which  is  an  output  of  Program  P63.  The  contents  of  this  file  are  printed 
on  the  Financial  Analysis  Summary  Report  FA02.  Various  calculational  runs  can 
be  stored  and  given  a unique  sequence  number. 

The  format  of  this  file  is  as  follows: 


Pos  1-2 


Pos  3 


Calculational  Sequence  Run  Number  - Up  to  100 
sequences  (00-99)  can  be  stored,  the  00  sequence 
will  be  the  baseline  run  from  the  data  stored  in 
the  CMR  files. 

Sequence  Number  (0-5)  relating  to  sequence  of 
project  number  in  ALTP  file  being  compared  with 
the  primary  project.  A 0 sequence  number  is  the 
data  on  the  primary  project. 


1 11-31 


TAEG  REPORT  NO.  40 


PAGE  NUMBER  : 1 

SOURCE  CARO  LISTING  FOR 
••ALTP 

••ALTP 

PNI24  PN125  PNI26  PNI27  PNI28 

FIGURE  111-16.  ALTP  FILE  EXAMPLE 
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Pos 

4-  8 

Project  Identifier;  e.g.,  PN123  for  primary 
project  (0  in  Pos  3),  PN124  for  first  alter- 
native project  (1  in  Pos  3),  etc. 

Pos 

9-  10 

Project  Life  in  Years 

Pos 

11-  16 

Total  Present  Value  (PV)  of  New  Investment  ($K) 

Pos 

17-  22 

Total  PV  of  Recurring  Cost/Savings  ($K) 

Pos 

23-  28 

Net  Present  Value  ($K) 

Pos 

29-  30 

Discount  Rate  (Percent) 

Pos 

31-  34 

Savings/Investment  Ratio 

Pos 

35-  37 

Rate  of  Return  on  Investment 

Pos 

38-  43 

Uniform  Annual  Cost 

Pos 

44-  45 

Blank 

Pos 

46-  255 

Descriptive  data  indicating  run  conditions.  Pos 

46-53  of  the  first  record  in  every  sequence  (0  in 
Col  3)  will  have  the  date  of  the  run  stored  as 
MM/DD/YY. 


Project  Data  Base  Application  Summary.  The  files  outlined  for  the  project  data 
base  will  contain  data  associated  with  a single  analysis  of  the  proposed  innova- 
tion. Data  input  and  computational  changes  can  be  made  and  stored  until  the 
assessor  is  satisfied  with  the  results  at  that  point  in  time.  No  attempt  is 
made  to  store  historical  data,  although  this  can  be  accomplished  merely  by  copying 
each  of  the  project  files  under  a different  identifier.  The  intent,  however,  is 
to  produce  a project  report  reflecting  the  assessment  results  at  a point  in  time. 

If  subsequent  reassessment  is  desirable  because  factors  or  conditions  have  changed, 
new  data  input  and  computational  results  will  be  written  over  previously  stored 
data. 

ETAM  PROCESSING  SEQUENCE.  Two  major  processing  sequences  are  involved  in  the 
use  of  ETAM.  These  are  in  addition  to  the  processing  required  to  initially 
establish  the  project  data  base  through  card  input.  The  first  major  sequence 
is  Range-of-Effect.  In  Range-of-Effect,  the  assessor  identifies  search  descrip- 
tors which  will  be  matched  to  the  abbreviated  data  bases  in  order  to  discover 
entities;  i.e.,  courses,  vehicles,  and  job/tasks,  that  may  have  potential  appli- 
cation for  the  proposed  innovation.  Initial  search  descriptors  can  be  entered 
into  the  REC,  REV,  and  REJ  files  via  card.  The  assessor  can  then  interactively 
refine  these  search  descriptors  based  upon  search  results.  If  the  matched 
entities  do  not  appear  appropriate,  the  assessor  can  interactively  add  or  delete 
to  create  a final  entity  list  which  will  be  the  basis  for  obtaining  additional 
data  for  the  assessment  process.  The  output  of  the  Range-of-Effect  processing 
sequence  is  a list  of  entities  about  which  additional  data  must  be  collected. 

The  entity  lists,  along  with  preprinted  data  collection  forms,  are  forwarded  to 
the  appropriate  Navy  source  in  order  to  obtain  these  additional  data.  The  data 
collected  are  returned,  punched  into  cards,  and  loaded  into  extract  files  EXC, 

EXV , EXJ,  and  EXD.  This  process  completes  the  Range-of-Effect  portion  of  ETAM. 
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The  second  major  ETAM  processing  sequence  involves  an  assessment  of  the  innovation. 
The  Assessment  sequence  involves  the  application  of  the  apparent  benefit  pattern, 
accruing  from  the  innovation,  to  the  quantitative  and  qualitative  variables  which 
will  be  evaluated  during  the  Assessment  processing  sequence.  With  the  benefit 
pattern  established,  cost  models  can  be  run  and  the  results  stored  for  subsequent 
use  by  either  the  Decision  Analysis  or  the  Financial  Analysis  programs.  During 
the  Decision  Analysis  part  of  the  Assessment  program,  the  assessor  interactively 
assesses  certain  decision  tree  parameters  in  order  to  calculate  decision  variables 
for  potential  decision  routes.  In  the  Financial  Analysis  part  of  the  Assessment 
program,  certain  financial  factors  are  calculated  which  gives  the  decision  maker 
a different  perspective  in  evaluating  the  innovation  than  was  gained  from  the 
Decision  Analysis  program.  These  two  programs  comprise  the  major  part  of  the 
Assessment  processing  sequence.  Following  is  a brief  description  of  each  of  the 
steps  which  the  assessor  can  take  as  a part  of  the  ETAM  innovation  evaluation 
process. 

Figure  1 11-17  provides  a descriptive  summary  of  the  various  ETAM  steps.  The 
figure  includes  references  to  project  data  base  files  and  programs  which  are 
an  integral  part  of  the  ETAM  processing  sequence. 

Create  Project  Files.  At  the  inception  of  the  project,  as  much  information  as 
is  available  should  be  entered  into  the  project  file.  Thirteen  of  the  Project 
Data  Base  files  can  be  created  from  punched  cards  in  batch  mode.  They  can 
subsequently  be  revised  by  reentering  the  new  file  content  via  card,  or  inter- 
actively modifying  any  file  using  the  National  CSS  Edit  program. 

o Operation:  Batch  or  Interactive 

o Files  : ID,  REC,  REV,  REJ,  EXD,  BPT,  RKP,  RRPJ,  RRPK,  VQAL,  VARF, 

SCEN,  ALTP 

o Programs  : PI  7 

Review  Project  Files.  Project  descriptive  data  is  reviewed  by  the  assessor. 

This  could  include  reference  to  previously  printed  project  data  as  well  as 
to  information  on  alternative  projects.  This  is  not  necessarily  limited  to 
a review  of  the  ID  file. 

o Operation:  Manual 

o Files  : ID,  REC,  REV,  REJ,  BPT,  RKP,  RRPJ,  VQAL,  SCEN,  ALTP 

o Programs  : PI  to  print  Project  Data  Base  files  (if  not  already 

available) . 

Refine  Search  Descriptors.  The  search  descriptors  initially  identified  for  the 
innovation  can  be  refined  interactively  by  the  assessor  in  order  to  create  a 
final  pattern  used  to  search  and  obtain  additional  information  in  the  abbreviated 
data  bases. 

o Operation:  Interactive 

o Files  : REC,  REV,  REJ,  SARG 

o Programs  : P2 

Search.  The  results  of  the  refinement  of  the  search  parameters  in  files  REC, 

REV,  and  REJ  are  used  to  search  the  abbreviated  data  bases  (Course,  Job/Task, 
and  Vehicles)  for  matching  descriptors.  The  results  are  stored  in  a temporary 
file  for  subsequent  refinement  by  the  assessor. 
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o Operation:  Interactive 

o Files  : SARG,  ABBREVIATED  DATA  BASE,  RESU 

o Programs  : P3 

Refine  Searched  Items.  The  entities  in  the  abbreviated  data  bases  which  match 
the  search  parameters  are  reviewed  and  refined  by  the  assessor.  This  is  accom- 
plished through  interactive  reference  to  the  temporary  file  where  the  search 
results  are  stored.  Additions  and  deletions  made  at  this  time  are  flagged  and 
the  final  results  are  (1)  stored  in  project  file  REE,  and  (2)  printed  for  trans- 
mittal to  other  Navy  personnel  for  completion. 

o Operation:  Interactive 

o Files  : RESU,  REE 

o Programs  : P5A  prints  the  search  results 

P5B  accesses  the  search  results  for  editing. 

Prepare  Extract  Forms.  The  final  results  of  the  search  and  subsequent  inter- 
active refinement  produce  the  entities  potentially  impacted  by  the  proposed 
innovation.  Before  the  assessment  process  can  begin,  additional  data  will 
generally  be  required.  Printed  extract  forms  onto  which  the  additional  infor- 
mation can  be  written  are  prepared  and  forwarded  to  the  appropriate  Navy  per- 
sonnel for  completion. 

o Operation:  Interactive  and  Manual 

o Files  : REE 

o Programs  : P5C 

Complete  Extract  Forms.  Lists  of  entities  obtained  through  the  search  and 
subsequent  edit  will  be  forwarded  to  responsible  Navy  functions  where  additional 
data  regarding  costs,  usage,  etc.,  can  be  obtained.  Training  information  source 
would  be  Chief  Naval  Education  and  Training  for  NITRAS  data,  vehicle  data  source 
would  be  the  Naval  Training  Equipment  Center  for  Training  Device  Statistical 
Data,  and  the  job/task  source  would  be  the  Bureau  of  Personnel  for  Navy  Occupa- 
tional Task  Analysis  Program  data. 

o Operation:  Offline  Batch  or  Manual 

o Files  : None 

o Programs  : Tape  Utility,  Report  Generator,  etc. 

Update  Extract  Files.  Additional  information  entered  on  the  printed  extract 
forms  is  keypunched  and  loaded  into  the  project  files  EXC,  ECV,  and  EXJ. 

Where  certain  data  are  not  available,  an  entry  will  be  made  in  the  EXD  file 
to  indicate  the  appropriate  default  condition. 

o Operation:  Batch 

o Files  : EXC,  EXV,  EXJ,  EXD 

o Programs  : P10,  PI  1 , P 1 2 , P17 

Review  Benefit  Description.  A pattern  of  potential  benefits  associated  with  the 
introduction  of  the  proposed  innovation  will  have  been  described  and  stored  in 
project  file  BPT.  The  assessor  reviews  these  data  in  preparation  for  modifying 
appropriate  cost  model  variables. 

o Operation:  Manual 

o File  : BPT 

o Programs  : PI  to  print  BPT  file  (if  not  already  available). 
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Apply  Benefit  to  Costs.  The  descriptive  benefit  pattern  must  be  quantitatively 
defined  by  the  assessor  to  the  appropriate  cost  model.  This  is  accomplished  by 
interactively  operating  upon  the  cost  model  input  variables  prior  to  nning  the 
model(s).  This  step  results  in  the  creation  of  a temporary  file  containing  cost 
model  input  data. 


o 

o 

o 


Operation:  Interactive 

Files  : EXC,  EXV,  EXJ,  COST  MODEL  INPUT 

Programs  : P51 


Run  Cost  Models.  The  assessor  now  invokes  the  appropriate  cost  model (s)  and 
the  model  outputs  are  printed  and  stored  in  project  file  CMR. 


o 

o 

o 


Operation:  Interactive 

Files  : COST  MODEL  INPUT,  EXD,  CMR 

Programs  : P51 , P52,  P53,  P54 


Review  Risk  Profiles.  The  next  major  ETAM  stage  is  assessment  and  introduction 
of  risk  factors.  The  assessor  prepares  for  this  part  of  the  process  by  reviewing 
the  contents  of  project  file  RKP  which  contains  risk  profiles  for  each  of  the 
risk  categories;  e.g.,  state-of-the-art,  technical  support,  etc. 


o Operation:  Manual 

o Files  : RKP 

o Programs  : PI  to  print  RKP  file  (if  not  already  available). 


Review  Risk  Reduction  Projects.  A number  of  projects  will  have  been  structured 
to  reduce  the  potential  risks  in  each  of  the  risk  categories  associated  with  the 
implementation  and  use  of  the  innovation.  The  assessor  reviews  these  in  associ- 
ation with  the  '•isk  profiles  previously  identified. 


o Operation:  Manual 

o Files  : RKP,  RRPJ 

o Programs  : PI  to  print  RKP  and  RRPJ  files  (if  not  already 

available) . 

Group  Risk  Reduction  Projects.  All  of  the  risk  reduction  projects  identified  may 
not  be  economically  feasible  to  undertake.  A better  cost-benefit  effect  may  be 
obtained  by  selecting  a group  of  the  risk  projects  for  implementation.  Available 
resources  will  undoubtedly  be  a major  determinant  of  the  number  of  projects  under- 
taken. The  assessor  reviews  the  risk  profiles  and  risk  reduction  projects  (project 
files  RKP  and  RRPJ)  and  identifies  various  packages  of  projects  which  appear 
reasonable  to  undertake.  Interactively,  the  groupings  can  be  made  to  which  the 
assessor  must  attach  certain  probabilities  and  probability  ranges  (confidence 
ranges).  The  first  probability  and  confidence  range  to  be  estimated  is  associated 
with  the  expected  success  of  the  project  package  itself.  If  unique  talent  is 
required  to  undertake  the  projects,  this  probability  would  be  low.  The  second 
probability  estimate  is  associated  with  the  expected  success  in  implementing  the 
innovation.  An  implementation  success  probability  will  be  estimated  considering 
no  risk  reduction  project  package  is  undertaken.  Assumedly,  each  package  of 
projects  identified  will  improve  this  probability  estimate  to  some  degree.  A 
confidence  range  is  also  estimated.  The  third  probability  estimated  is  that  of 
success  in  obtaining  acceptance  by  the  using  system.  Again,  this  probability 
will  be  estimated  under  the  assumption  of  no  risk  reduction  project  packages. 

Then,  a new  probability  will  be  estimated  for  each  package  considering  its  effect 
upon  the  using  system's  acceptance  of  the  innovation.  This  estimate  will  include 
a confidence  range. 
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o Operation:  Interactive 

o Files  : RKP , RRPJ,  RRPK 

o Programs  : P56  (P59  may  be  called) 

Review  Qualitative  Variables  Affected.  The  descriptions  of  the  qualitative 
variables,  e.g.,  morale,  safety,  etc.,  are  reviewed  by  the  assessor  in  preparation 
for  the  scaling  operation.  These  descriptions  are  stored  in  project  file  VQAL. 

o Operation:  Manual 

o Files  : VQAL 

o Programs  : PI  to  print  VQAL  file  (if  not  already  available). 

Scale  Qualitative  Variables.  In  this  step,  the  assessor  develops  relative  values 
for  each  of  the  relevant  qualitative  variables  as  well  as  the  cost/savings  variable 
previously  calculated.  This  is  an  interactive  process,  with  the  assessor  being 
prompted  into  estimates  of  the  present  and  potential  states  at  each  outcome  of 
each  relevant  variable.  An  importance  value  is  assigned  to  each  attribute  (variable) 
which  permits  subsequent  relevant  values  (e.g.,  equivalent  dollars)  to  be  attached  to 
each  attribute  at  each  possible  outcome. 

o Operation:  Interactive 

o Files  : CMR,  VQAL,  VARF 

o Programs  : P58 

Review  Outcome  Scenarios.  The  assessor  must  now  visualize  the  effects  of  intro- 
ducing the  innovation  in  each  of  the  potential  outcome  situations,  i.e.,  (1)  it 
is  implemented  and  accepted  by  the  using  system,  (2)  it  is  implemented  but  fails 
to  gain  acceptance  by  the  using  system,  (3)  it  fails  to  be  implemented  after  being 
accepted,  or  (4)  it  is  rejected.  These  scenarios  if  previously  developed  are 
described  in  project  file  SCEN. 

o Operation:  Manual 

o Files  : SCEN  1 

o Programs  : PI  to  print  SCEN  file  (if  not  already  available). 

Create  Outcome  Values.  An  equivalent  dollar  value  or  utility  is  now  developed 
for  each  of  the  four  decision  tree  outcome  states.  This  is  performed  inter- 
actively by  the  assessor  using  costs  from  previous  model  runs,  liability  costs 
or  shutdown  costs  developed  for  each  outcome  where  the  innovation  is  accepted 
but  does  not  succeed  in  either  implementation  or  using  system  acceptance,  point 
estimates  and  importance  estimates  identified  in  the  scaling  of  qualitative 
variables  step,  and  judgments  developed  during  the  interactive  process.  The  final 
outcome  totals  will  be  used  in  the  folding  back  of  the  decision  tree  to  develop 
the  decision  variables. 

o Operation:  Interactive 

o Files  : CMR,  VARF,  SCEN 

o Programs  : P58 

Assign  Probabilities.  The  remaining  input  to  the  decision  tree  is  the  finalized 
set  of  probabilities  associated  with  each  mode  of  the  tree.  These  were  initially 
developed  in  conjunction  with  the  step  in  which  risk  reduction  projects  were 
grouped  into  various  packages.  The  assessor  on  an  interactive  basis  reviews  and 
refines  these  probabilities. 


* 
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o Operation:  Interactive 

o Files  : VARF,  SCEN 

o Programs  : P59 

Run  Decision  Analysis.  The  assessor  invokes  the  decision  tree  evaluator  which 
computes  the  expected  value  for  each  possible  decision  path.  The  computation 
simply  folds  back  the  tree  based  upon  the  outcome  values  and  the  node  probabilities. 
A decision  variable  is  calculated  for  the  decision  path  where  the  proposed  innova- 
tion is  accepted  without  performing  any  of  the  risk  reduction  projects,  as  well 
as  for  each  decision  path  involving  a group  or  package  of  risk  reduction  projects 
which  would  be  expected  to  be  undertaken.  The  results  of  the  decision  tree  evalu- 
ator program  run  are  printed  and  also  stored  in  the  project  file  TRER. 

o Operation:  Interactive 

0 Files  : CMR,  RRPK,  VARF,  TRER 

o Programs  : P60 

Review  Alternative  Projects.  If  alternatives  to  the  primary  innovation  have  been 
proposed,  they  should  be  reviewed  by  the  assessor  at  this  time.  It  is  assumed  that 
most  of  the  previous  steps  will  have  been  considered  for  each  of  the  alternative 
proposals.  If  calculation  of  decision  tree  variables  for  an  alternative  proposal 
indicates  in  of  itself  it  should  be  rejected;  it  would  not  generally  be  considered 
in  any  comparative  financial  analysis  with  the  primary  proposal  or  other  promising 
alternative  proposals.  The  identifiers  for  alternative  proposals  to  be  reviewed 
at  this  time  are  stored  in  project  file  ALTP. 

o Operation:  Manual 

o Files  : ALTP,  OTHER  PROJECT  DATA  BASE  FILES  REFERENCED  BY 

ALTP 

o Programs  : PI  to  print  ALTP  and  other  PROJECT  DATA  BASE  files 

(if  not  already  available). 

Run  Financial  Analysis.  This  step  takes  the  incremental  cost/savings  data 
associated  with  each  proposed  innovation  (primary  and  alternatives)  and  calcu- 
lates a set  of  financial  measures  (e.g.,  rate  of  return,  etc.)  when  comparing 
the  proposal  to  the  existing  system.  Marginal  measures  of  return  can  be  cal- 
culated comparing  two  proposals.  The  assessor  sets  up  the  run  conditions  and 
invokes  the  financial  analysis  routine.  The  results  of  these  program  runs  are 
stored  in  project  file  FINR. 

o Operation:  Interactive 

o Files  : ALTP,  CMR  (from  Primary  and  Alternative  projects),  FINR 

o Programs  : P62,  P63 

Processing  Step  Summary.  The  ETAM  steps  have  described  more  or  less  in  a logical 
sequence  for  performance  by  the  assessor.  A major  intent  in  developing  the  ETAM 
procedures  was  to  foster  an  efficient  as  well  as  effective  decision  process. 

This  requires  that  the  steps  previously  outlined  be  performed  iteratively  with 
increasing  amounts  of  data,  the  collection  of  which  appears  justified  at  each 
iterative  stage.  Therefore,  the  assessor  should  take  advantage  of  the  capability 
for  randomly  assessing  any  of  the  routines  and  running  them  given  some  level  of 
valid  data  inputs  can  be  developed. 
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ETAM  PROGRAM  FUNCTIONS 

The  ETAM  process  sequence  previously  described  is  a composite  of  manual,  com- 
puter batch,  and  computer  interactive  steps  which  require  system  files,  project 
data  base  files,  and  computer  programs.  The  Project  Data  Base  files  were  out- 
lined in  the  preceding  section.  The  programs  which  were  developed  for  the  ETAM 
Range-of-Effect  function  are  summarized  here,  and  documented  in  detail  in  Appendix 
D.  This  section  outlines  the  design  requirements  for  completing  the  programming 
development  of  the  ETAM  Assessment  function. 

ETAM  CONTROL  EXEC.  ETAM  programs  can  be  run  through  individual  EXEC's  which  call 
each  program.  For  example,  entering  RUNP1  PN123  at  the  terminal  would  call  the 
RUNP1  EXEC  supplying  it  with  the  parameter  PN123  as  the  Project  Identifier.  The 
RUNP1  EXEC  would  in  turn  call  Program  PI  which  would  permit  the  files  in  the 
Project  Data  Base  to  be  printed.  However,  the  ETAM  processing  sequence  can  be 
controlled  through  the  ETAM  EXEC  which  is  called  by  entering  ETAM  at  the  terminal. 
The  ETAM  EXEC  prompts  the  user  in  the  selection  of  the  project  and  programs  to 
be  accessed.  The  logic  of  the  ETAM  EXEC  is  shown  in  Figure  1 1 1 - 1 8 . Each  of  the 
terminal  displays  is  numbered  1 through  8.  The  content  of  each  display  is  shown 
in  Figure  1 1 1 - 1 9 . Since  programs  for  the  Assessment  function  are  not  developed, 
the  user  will  have  displayed  "PROGRAM  NOT  INSTALLED"  anytime  an  attempt  is  made 
to  call  MODEL,  RISK,  DECISION  or  FINANCIAL  (Assessment-Select  Option,  Disp  #7' 
in  Figure  1 1 1- 1 9) . 

Besides  permitting  selection  of  the  two  primary  ETAM  functions;  i.e.,  Range-of- 
Effect  or  Assessment,  the  ETAM  EXEC  1)  checks  for  existence  of  the  Project 
Identifier  inputted  by  the  user,  2)  permits  generation  of  a new  Project  Data 
Base  by  calling  Program  P17,  3)  permits  printing  of  the  Project  Data  Base  by 
calling  Program  PI,  or  4)  displays  instructions  for  the  user. 

Project  Identifier  Existence.  When  the  ETAM  EXEC  is  called  by  entering  ETAM, 
a Project  ID  is  requested.  The  existence  of  the  ID  entered  is  checked  through 
a system  procedure  within  the  ETAM  EXEC  which  looks  for  an  ID  File  Type  with 
the  Project  ID  as  the  File  Name.  The  system  requires  that  all  Project  ID's 
begin  with  a non-numeric  character,  however,  the  ETAM  convention  is  to  begin 
each  Project  ID  with  the  characters  PN  and  follow  it  with  three  numeric  desig- 
nators; e.g.,  PN123.  No  system  checks  will  ensure  this  format  is  followed  by 
the  user,  although  the  EXEC  prompting  message  provides  this  format  in  the 
example.  Thus,  if  an  ID  file  exists  with  the  entered  Project  ID,  the  user  can 
continue  through  the  ETAM  evaluation  process.  If  the  Project  ID  does  not  already 
exist,  the  user  will  be  given  the  option  of  generating  a new  Project  Data  Base 
using  Program  P17. 

Program  P17  ( Project  Generati on ) . In  order  to  generate  a new  Project  Data  Base, 
a LOAD  File  Type  with  the  new  Project  ID  as  the  File  Name  must  exist  in  the 
user's  workspace.  It  may  have  been  entered  through  an  OFFLINE  READ  of  the 
cards  constituting  the  project  files,  or  it  could  have  been  created  from  the 
terminal  using  the  NCSS  Edit  Procedures. 

The  first  card  of  the  LOAD  file  must  contain  INIT  in  the  first  four  positions. 

The  second  card  of  the  LOAD  file  must  contain  **ID  in  the  first  four  positions. 

The  third  card  must  contain  the  Project  ID  anywhere  within  the  first  ten  positions, 
and  should  contain  the  name  of  the  new  project  in  Pos  11-80.  Figure  1 1 1 -20 
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DISP  # 
1 
2 

3 


DISPLAY  CONTENT 

ENTER  PROJECT  IDENTIFIER,  E.G.,  PN123  (5  CHARACTERS) 

PROJECT  DOES  NOT  EXIST. 

GFNERATE  NEW  FILE?  ? ? 

1)  YES 

2)  NO 

ENTER:  1 OR  2 

SELECT  OPTION. 

1)  ROE 

2)  ASSESS 

3)  PRINT 

4)  NEW  PROJECT 

5)  INSTRUCTIONS 

6)  QUIT 

ENTER:  1 ...  6 

ETAM  INSTRUCTIONS 

OPTIONS  I THRU  6 PERMIT  SELECTION  OF  THE  FOLLOWING  ETAM 
FUNCTIONS. 

1 WILL  CAUSE  ENTRY  INTO  THE  ETAM  RANGE-OF-EFFECT  FUNCTION 
FOR: 

RUNNING  DESCRIPTOR  EDIT  PROGRAM 
SEARCHING  ABBREVIATED  DATA  BASES 
PRINTING  SEARCH  RESULTS 
RUNNING  SEARCH  EDIT  PROGRAM 
RUNNING  PRINT  EXTRACT  PROGRAM 

2 WILL  CAUSE  ENTRY  INTO  THE  ETAM  ASSESSMENT  FUNCTION  FOR: 

SETTING  UP  MODEL  INPUTS  FOR  MODEL  RUN 
PACKAGING  RISK  REDUCTION  PROJECTS 
RUNNING  DECISION  ANALYSIS  PROGRAMS 
RUNNING  FINANCIAL  ANALYSIS  PROGRAMS 

3 WILL  PERMIT  PRINTING  OF  PROJECT  DATA  BASE  FILES 

4 WILL  CAUSE  A RETURN  TO  THE  BEGINNING  OF  THE  ETAM  CONTROL 

EXEC  SO  THAT  A DIFFERENT  PROJECT  ID  CAN  BE  ENTERED 

5 WILL  CAUSE  THESE  ETAM  INSTRUCTIONS  TO  BE  REPRINTED 

6 WILL  END  ETAM  EXEC  CONTROL  AND  RETURN  TO  NCSS  CONTROL 


FIGURE  1 11-19.  ETAM  CONTROL  EXEC  DISPLAY  PATTERNS 
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DISP  # DISPLAY  CONTENT 


5 RANGE-OF-EFFECT  - SELECT  OPTION. 

1)  EDIT  DESCRIPTOR 

2 SEARCH 
9)  PRINT  SEARCH 

4)  EDIT  SEARCH 

5)  PRINT  EXTRACT 

6)  QUIT 

ENTER:  1 ...  6 

6 CONTINUE  ROE  TO  SELECT  OTHER  ROUTINES?  ? ? 

1)  YES 

2)  NO 

ENTER:  1 OR  2 

7 ASSESSMENT  - SELECT  OPTION. 

1 ) MODEL  f 

2)  RISK 

3)  DECISION 

4)  FINANCIAL 

5)  QUIT 

ENTER:  1 ...  5 

8 CONTINUE  ASSESS  TO  SELECT  OTHER  ROUTINES?  ? ? 

1)  YES 

2)  NO 

ENTER:  1 OR  2 


FIGURE  1 1 1- 1 9 . ETAM  CONTROL  EXEC  DISPLAY  PATTERNS  (Cont'd) 
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kid. 43. 16  RUMP  1 7 PM  1 2 3 

PROJECT  NAMED  PM123  DOES  MOT  YET  EXIST, 
IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 


OUTPUT  TO  GO  TO  TERMINAL,  OFFLINE  PRINTER,  bOTH,  OR  PUIT? 

ENTER  TERM,  PRTR,  bOTH,  OR  QUIT 

PRTR 

^execution: 

PROJECT  FILE  INITIALIZATION  PROGRAM  IS  STARTING 

LOAD  OF  PROJECT  FILETYPE  : ID  - PROJECT  DESCRIPTION 

LOAD  OF  PROJECT  FILETYPE  : REC  - COURSES  ROE  SEARCH  ARGUMENTS 

LOAD  OF  PROJECT  FILETYPE  : BPT  - BENEFIT  PATTERN 

LOAD  OF  PROJECT  FILETYPE  : RKP  - RISK  PROFILE 

LOAD  OF  PROJECT  FILETYPE  : RRPJ  - RISK  REDUCTION  PROJECTS 

LOAD  OF  PROJECT  FILETYPE  : SCEN  - SCENARIOS 

LOAD  OF  PROJECT  FILETYPE  : RRPK  - RISK  REDUCTION  PACKAGES 

LOAD  OF  PROJECT  FILETYPE  : VQAL  - VARIABLES  QUALITATIVE 

LOAD  OF  PROJECT  FILETYPE  : VARF  - VARIABLE  REFERENCES 

LOAD  OF  PROJECT  FILETYPE  : ALT P - ALTERNATE  PROJECTS 


FILE 

: REJ 

HAS 

BEEN 

INITIALIZED 

WITH 

ONE 

(1) 

BLANK 

RECORD 

FILE 

: REV 

HAS 

BEEN 

INITIALIZED 

WITH 

ONE 

Cl) 

blank 

RECORD 

FILE 

: EXD 

HAS 

BEEN 

initialized 

WITH 

ONE 

Cl) 

BLANK 

RECORD 

::::  ALL  DATASETS  INITIALIZED  ::5: 


::::  TOTAL  NUMBER  INPUT  CARDS  READ  : 46  1 :::: 

PROJECT  FILE  INITIALIZATION  PROGRAM  IS  NOW  TERMINATING 


FIGURE  111-20.  EXAMPLE  OF  PROGRAM  P17 
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shows  an  example  run  of  Program  P17  which,  in  this  case,  was  called  by  the 
RUNP17  EXEC  with  the  new  Project  ID  as  a parameter.  The  Project  ID  parameter 
would  have  been  passed  automatically  if  the  RUNP17  EXEC  were  called  by  the 
ETAM  EXEC. 

Thirteen  file  types  can  be  loaded  using  Program  P17.  These  are  listed  in 
Figure  1 1 1-20.  Any  of  these  files  for  which  data  do  not  exist  in  the  LOAD 
File  Type  will  be  automatically  initialized  at  this  time  by  Program  PI 7.  When 
data  are  available  for  the  missing  files,  or  previously  created  files  are  to 
be  modified.  Program  PI 7 will  be  used  again.  The  difference  between  this 
process  and  the  initial  Project  Data  Base  generation  procedure  is  that  the  LOAD 
File  Type  for  an  update  will  not  contain  the  INIT  card. 

Program  PI 7 gives  the  user  the  option  of  listing  the  new  Project  Data  Base  con- 
tents at  either  the  terminal  or  the  printer,  or  both.  The  user  should  obtain  the 
initial  Project  Data  Base  listing  in  this  manner. 

Program  PI  (Project  Data  Base  Print).  Program  PI  permits  the  user  to  list  any 
one  or  all  of  twenty-one  of  the  twenty- two  files  in  the  Project  Data  Base  (the 
SARG  file  is  not  listed).  The  output  can  be  directed  to  either  the  terminal 
or  offline  printer,  or  both.  While  the  option  to  print  ALL  Project  Data  Base 
files  is  available  to  the  user,  it  can  not  be  used  until  all  the  Project  Data 
Base  files  printed  by  Program  PI  have  been  opened.  Figure  1 1 1 -21  shows  and  example 
run  of  Program  PI  as  called  by  the  RUNP1  EXEC.  When  ETAM  is  entered  through  the 
ETAM  EXEC,  running  of  Program  PI  is  one  of  the  select  options. 

ETAM  Instructions.  The  instructions  which  will  be  displayed  to  the  user  were 
shown  in  Figure  1 11-19,  Disp  #4. 

ETAM  RANGE-OF-EFFECT.  The  Range-of-Effect  is  a major  ETAM  function  which 
attempts  to  identify  potential  applications  for  a proposed  innovation.  The 
methodology  is  based  upon  a capability  of  assigning  descriptors  to  the  innova- 
tion which  serve  as  search  parameters  to  data  bases  having  courses,  vehicles, 
and  job/tasks  to  which  the  same  set  of  descriptors  have  been  associated.  The 
descriptors  were  developed  in  such  a way  that  a match  between  a search  descrip- 
tor assigned  to  the  innovation,  and  an  entity  index  descriptor  assigned  to  a 
course,  for  example,  should  represent  a potential  application  for  the  innovation. 
The  descriptor  sets  are  discussed  in  Appendix  A of  this  report.  The  purpose  of 
this  section  is  to  describe  the  programs  for  accomplishing  the  major  Range-of- 
Effect  objectives,  which  are  to: 

1.  Assign  descriptors  to  the  proposed  innovation  based  upon  the 
descriptor  taxonomies  outlined  in  Appendix  A. 

2.  Search  the  Abbreviated  Data  Bases  for  courses,  vehicles  or 
job/tasks  (whichever  are  appropriate  to  the  potential  applica- 
tion of  the  innovation),  and  select  a set  of  entities  which 
match  the  search  parameter  logic. 

3.  Print  a list  of  the  selected  entities,  then  refine  the  list  by 
adding  or  deleting  items  based  upon  a broader  perspective  of 

the  entity  characteristics  than  is  embodied  in  the  index  descrip- 
tors, and  finally  print  the  list  of  entities  requiring  additional 
dc'.ta  for  the  ETAM  Assessment  phase. 


j 


i 
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CSS 

,19.18.59  RUNP1  PN123 

PROJECT  NAMED  PN123  ALREADY  EXISTS, 

IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 

S$$$$ 

GENERAL-PURPOSE  PROJECT  FILE  PRINT 

PROGRAM  IS  NOW  STARTING... 

;j::  ENTER  FILETYPE  FOR  OUTPUT,  MENU,  OR  QUIT 

_MENU 

PROJECT  FILETYPES  FOR  OUTPUT  ARE  : 

ID  - PROJECT  DESCRIPTION 

EXD  - EXTRACT  DEFAULTS 

UP T - UENEFIT  PATTERN 

RKP  - RISK  PROFILE 

RRPJ  - RISK  REDUCTION  PROJECTS 

RRPK  - RISK  REDUCTION  PACKAGES 

VQAL  - VARIABLES  QUALITATIVE 

VARF  - VARIABLE  REFERENCES 

SCEN  - SCENARIOS 

ALTP  - ALTERNATE  PROJECTS 

REC  - COURSES  ROE  SEARCH  ARGUMENTS 

REV  - VEHICLES  ROE  SEARCH  ARGUMENTS 

REJ  - TASKS  ROE  SEARCH  ARGUMENTS 

RE  SU  - ROE  TOTAL  SEARCH  RESULTS 

REE  - ROE  SEARCH  RESULTS 

EXC  - EXTRACT  DR  - COURSES 

EXJ  - EXTRACT  DB  - JOBTASKS 

E XV  - EXTRACT  DB  - VEHICLES 

CMR  - COST  MODEL  RESULTS 

TRER  - TREE  RESULTS 

FINR  - FINANCIAL  RESULTS 


FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI 
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::::  ENTER  F I LET  YPE  FOR  OUT  PUT , MENU,  OR  QUIT  ::- 
_REC 

OUTPUT  TO  GO  TO  TERMINAL,  OFFLINE  PRINTER,  BOTH,  OR  QUIT 

ENTER  TERM,  PRTR,  BOTH,  OR  QUIT 

_BOTH 

PAGE  NUMBER  : 1 

LISTING  OF  PROJECT  FILL  CONTENTS 

PN123  / REC  - COURSES  ROE  SEARCH  ARGUMENTS 

30  OPERATIONS 

31  MAINTENANCE 

40  EQUIPMENT  £ OBJECTS  USED:  REAL 
64  PROCEDURE  FOLLOWING 
-80  OR  I E NT AT  ION,  FAMILIARIZATION 

-81  TASK  NOMENCLATURE,  IDENTS,  LOCATIONS,  FACTS,  RULES 

82  TASK  FORMATS  AT  CONCEPTUAL  LEVEL 

83  PROCEDURES  AT  VERBAL  LEVEL  ONLY 

84  TASK  COMPONENTS  WITH  GUIDANCE 

85  ENTIRE  JOB-TASK  PROCEDURALLY  AT  BARELY  ACCEPTABLE  MASTER Y 

8b  HIGHLY  PROFICIENT  IN  WORK  CONTEXT 

A TOTAL  OF  11  RECORDS  ARE  PRESENT 
WITHIN  THIS  FILE 


FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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ENTER  FILETYPE  FOR  OUTPUT , MENU,  OR  QUIT  "5: 

_RESU 

OUTPUT  TO  GO  TO  TERMINAL,  OFFLINE  PRINTER,  BOTH,  OR  QUIT 
ENTER  TERM,  PRTR,  BOTH,  OR  QUIT 
BOTH 


PAGE  NUMBER  : 2 

LISTING  OF  PROJECT  FILE  CONTENTS 

PN123  / RE SU  - ROE  TOTAL  SEARCH  RESULTS 

THE  FOLLOWING  IS  ROE  RESULT  DATA  FOR  COURSES 
NUM  ORIGIN  CIN  CDP  COURSE  TITLE 


1 

DELETED 

A1020060 

NONE 

(NO  ABBREVIATED 

F 

ILE 

DATA 

AVA 

I L ABLE  ) 

2 

DELETED 

A1020093 

NONE 

C NO  ABBREVIATED 

F 

ILE 

DATA 

AVA 

I LADLE  ) 

3 

DELETED 

A1020095 

NONE 

(NO  ABBREVIATED 

F 

ILE 

DATA 

AVA 

I L ABLE  ) 

4 

ORIGINAL 

A1930050 

22  13 

INERT  NAV  PR  INC 

5 

ORIGINAL 

A24200 10 

6529 

IS  A 

6 

OR  I GINAL 

A4 120010 

6078 

EA-A 

7 

ORIGINAL 

A4 1200  10 

6287 

EA  A 

FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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8 

OR  I G I NAL 

A43100U 

10  3 5 

EOD  BASIC  NAVY 

y 

ORIGINAL 

A4310014 

10  3b 

EOD  RE  FR  NAVY 

10 

ORIGINAL 

A4330019 

2040 

MASTER  DIVQUAL 

u 

ORIGINAL 

A4  3 3002  5 

2082 

DIVER  FIRST 

12 

ORIGINAL 

A4910014 

204G 

RADIUM  REMOV  OPS 

13 

ORIGINAL 

A5510019 

0133 

CAR  HAND  BAS 

14 

ORIGINAL 

A5510027 

393  1 

UNREP  MECH/HYD 

1 3 

ORIGINAL 

A3510068 

0134 

FORK  LIFT  OPER 

lb 

ORIGINAL 

A5800016 

b0  5 3 

CTO  A 

17 

ORIGINAL 

Ab 5 10020 

5 399 

12  00  PS  I BT 

18 

ORIGINAL 

AO520050 

52  24 

OX  GENR  6L160P 

19 

ORIGINAL 

A6320068 

4b  3E 

SOLAR  GAS  TURBIN 

20 

OR  I G I NAL 

A67000  1 1 

3078 

WATCH  REPAIR 

2 1 

ORIGINAL 

A7 010027 

33  7 M 

WLLD/HPRES  PIPE 

22 

ORIGINAL 

A7  2 000  1 3 

4667 

UT-J 

23 

DELETED 

B3030051 

NONE 

CNO  ABBREVIATED  FILE  DATA  AVAILABLE) 

24 

ORIGINAL 

C 003722 

2887 

TA4.IF  FAM  PILOTS 

23 

ORIGINAL 

C 003722 

9007 

TA4UF  FAM  PILOTS 

2b 

OR  I G I NAL 

C 003722 

9009 

TA4JF  FAM  PILOTS 

FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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PAGE  NUMBER  : 3 

LISTING  OF  PROJECT  FILE  CONTENTS 

PN123  / RESU  - ROE  TOTAL  SEARCH  RESULTS 

THE  FOLLOWING  IS  ROE  RESULT  DATA  FOR  COURSES 
NUM  ORIGIN  CIN  CDP  COURSE  TITLE 


27 

ORIGINAL 

C 003722 

9010 

TA4JF  FAM  PILOTS 

28 

ORIGINAL 

C 003722 

9738 

TA4JF  FA M PILOTS 

29 

ORIGINAL 

C 2C3352 

25  76 

UH IN  A/C  FAM/P/ 

30 

ORIGINAL 

C 1003834 

7529 

AAS18  I NT  MAI 

31 

ORIGINAL 

C1023793 

7914 

A7AB  ATT  HEADING 

32 

ORIGINAL 

C 1023793 

7915 

A7AB  ATT  HEADING 

33 

ORIGINAL 

C 12  1 3*1 1 1 

346L 

AWM23  RADIO  FREQ 

34 

ORIGINAL 

C 12 1 30  1 1 

544L 

AWM23  RADIO  FREO 

35 

ORIGINAL 

C6003472 

340C 

E2C  WEA  SYS  FAM 

36 

ORIGINAL 

C 602  35  36 

34  ID 

54H6077  PROP  INT 

37 

ORIGINAL 

C 646  3 10  3 

7 365 

CVA/CV  ALW  SUPV 

38 

DELETED 

D 2C0011 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

39 

DELETED 

E 2A1001 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

40 

DELETED 

E 2A1301 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

41 

DELETED 

E 2A1803 

NONE 

(NO  ABBREVIATED  T 

ILE 

DATA 

AVAILABLE 

42 

DELETED 

E 2C0901 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

43 

DELETED 

E 2D0016 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

44 

DELETED 

E 2D0075 

NONE 

(NO  ABBREVIATED  F 

ILE 

DATA 

AVAILABLE 

A 


111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 

III-SO 


FIGURE 


TAEG  Rl PORT  NO.  40 


45 

DELETED 

E6010210 

NONE 

(NO 

ABBREVIATED  FILE 

DATA 

AVAILABLE) 

46 

DELETED 

E0461641 

NONE 

(NO 

ABBREVIATED  FILE 

DATA 

AVAILABLE ) 

47 

DELETED 

F7000010 

NONE 

(NO 

ABBREVIATED  FILE 

DATA 

AVAILABLE ) 

46 

DELETED 

H 2E3710 

NONE 

(NO 

ABBREVIATED  FILE 

DATA 

AVAILABLE) 

49 

DELETED 

J 1 300645 

NONE 

(NO 

ABBREVIATED  FILE 

DATA 

AVA IL ABLE ) 

50 

OR IGINAL 

J2210357 

5 38Q 

MULTI -THREAT  TNG 

51 

ORIGINAL 

J2 3 3020 3 

2637 

ADV 

EW  OP'S  CRSE 

52 

ORIGINAL 

J 2430974 

2 in  i 

IPC 

PAGE  NUMBER 

: 4 

LISTING  OF  PROJECT  FILE  CONTENTS 

PN12  3 

/ RE  SU 

- ROE 

TOTAL  SEARCH  RES 

ULTS 

THE  FOLLOWING  IS 

ROE  RESULT 

DATA 

FOR 

COURSES 

NUM 

ORIGIN 

C IN 

CDP 

COURSE  TITLE 

53 

DELETED 

J8000433 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

54 

ORIGINAL 

K2330066 

205  V 

SUB 

EW  OP -AD V 

55 

DELETED 

L1010024 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

56 

ORIGINAL 

L6610056 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE ) 

57 

ORIGINAL 

Ml  98 100E 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

56 

ORIGINAL 

N7010320 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

59 

ORIGINAL 

Q 2C0015 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

60 

OR  I GINAL 

83000010 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

6 1 

OR  I G I NAL 

83000012 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

tw  ADDED  A1010108  7 f> 5 WRA-4  CMB  MA 

A TOTAL  OF  62  RECORDS  ARE  PRESENT 
WITHIN  THIS  FILE 


FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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;{5!  ENTER  FILETYPE  FOR  OUTPUT,  MENU,  OR  QUIT  «« 

_REE 

OUTPUT  TO  GO  TO  TERMINAL,  OFFLINE  PRINTER,  BOTH,  OR  OUIT 
ENTER  TERM,  PRTR,  BOTH,  OR  QUIT 
BOTH 


* 


PAGE  NUMBER  : 5 

LISTING  OF  PROJECT  FILE  CONTENTS 

PN123  / REE  - ROE  SEARCH  RESULTS 


THE  FOLLOWING  IS 
NUM  ORIGIN 


1 ORIGINAL 

2 ORIGINAL 

3 ORIGINAL 

4 OR  I G INAL 

5 ORIGINAL 

6 ORIGINAL 

7 ORIGINAL 


ROE  RESULT  DATA 


C IN 

CDP 

A1930050 

2213 

A2420010 

6529 

A4  12  0010 

6078 

A4  120010 

6287 

A4310011 

10  35 

A43I0014 

1036 

A4330019 

2040 

FOR  COURSES 
COURSE  TITLE 


INERT  NAV  PR  INC 
IS  A 
EA-A 
EA  A 

EOD  BASIC  NAVY 
EOD  RE  FR  NAVY 
MASTER  DIVOUAL 


■ 


FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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8 ORIGINAL  A4330025  2082  DIVER  FIRST 

y ORIGINAL  A4910014  204G  RADIUM  REMOV  OPS 

10  ORIGINAL  A5510019  0133  CAR  HAND  BAS 

11  ORIGINAL  A5510027  3931  UNRLP  MECH/HYD 

12  ORIGINAL  A55 10)16 8 0134  FORK  LIFT  OPCR 

13  ORIGINAL  A5800016  b053  CTO  A 

14  ORIGINAL  A 6 5 1 0 0 2 0 5399  1200  PSI  BT 

15  ORIGINAL  A6520050  5224  OX  GENR  bL160P 

lb  ORIGINAL  A652006J  463E  SOLAR  GAS  TURBIN 

17  ORIGINAL  AG700011  3078  WATCH  REPAIR 

la  ORIGINAL  A7 0 1002 7 337M  WELD/HPRES  PIPE 

19  ORIGINAL  A72000 13  466Z  UT-J 

20  ORIGINAL  C 003722  2887  TA4.JF  FAM  PILOTS 

21  ORIGINAL  C 003722  9007  TA4JF  FAM  PILOTS 

22  ORIGINAL  C 003722  9009  TA4 J F FAM  PILOTS 

23  ORIGINAL  C 003722  9010  TA4JF  FAM  PILOTS 

24  ORIGINAL  C 003722  9738  TA4JF  FAM  PILOTS 

25  ORIGINAL  C 2C3352  257b  UH1N  A/C  FAM/p/ 

26  ORIGINAL  C 1003834  7529  AAS18  I NT  MAI 
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PAGE  NUMBER  : G 

LISTING  OF  PROJECT  FILE  CONTENTS 


PN123 

/ REE  - 

ROE 

SEARCH  RESULTS 

THE  FOLLOWING  IS 
NUM  ORIGIN 

ROE  RESULT 
C IN 

DATA 
CD  P 

FOR  COURSES 
COURSE  TITLE 

27 

ORIGINAL 

C1023793 

7914 

A7AB  ATT  HEADING 

2b 

ORIGINAL 

C 1023793 

7915 

A7AB  ATT  HEADING 

29 

ORIGINAL 

C 1 2 1 30 1 1 

34GL 

AWM23  RADIO  FREQ 

30 

ORIGINAL 

C 12 130 1 1 

544L 

AWM23  RADIO  FREO 

31 

ORIGINAL 

C6003472 

34  0C 

E2C  WEA  SYS  FAM 

32 

ORIGINAL 

CG02353G 

34  ID 

54HG077  PROP  INT 

33 

ORIGINAL 

C6463103 

7 3G5 

CVA/CV  ALW  SUPV 

34 

ORIGINAL 

J2210357 

5 38Q 

MULT  I -THREAT  TNG 

35 

ORIGINAL 

J2330203 

2637 

ADV  EW  OP'S  CR5E 

3b 

OR  I G I NAL 

J2430974 

2181  . I PC 

37 

ORIGINAL 

K233006G 

205 Y SUB  EW  OP-ADV 

3b 

OR 

I GINAL 

LG61005G 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

39 

OR 

I G INAL 

M19b  100E 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

40 

OR 

I GINAL 

N 7010320 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

41 

OR 

I GINAL 

Q 2C0015 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

42 

OR 

I GINAL 

83000010 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

43 

OR 

I GINAL 

83000012 

NONE 

(NO 

ABBREVIATED 

FILE 

DATA 

AVA’LABLE) 

44 

ADDED 

A1010108 

7654 

(NO 

ABBREVIATED 

FILE 

DATA 

AVAILABLE) 

A TOTAL 

OF  44 

RECORDS 

ARE 

PRESENT 

WITHIN  THIS  FILE 


LNTFR  FILLTYPE  FOR  OUTPUT,  MENU,  OR  QUIT  «« 
QUIT 


PROJECT-FILL  PRINT  PROGRAM  IS  NOW  TERMINATING 


FIGURE  111-21.  EXAMPLE  OF  PROGRAM  PI  (Cont'd) 
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The  programs 


which  were  developed  to  accomplish  these  objectives  are: 


P2  - Interactive  Search  Designator  Editor 
P3  - Range-of-Effect  (ROE)  Search  Module 
P5A  - Print  of  Raw  ROE  Results  File 
P5B  - Interactive  Search  Results  Editor 
P5C  - Print  of  ROE  Search  Results  for  Extract 


Since  these  programs  are  developed  and  documented  in  Appendix  D of  this  report, 
only  the  user-related  functions  of  the  programs  are  outlined  here. 

Program  P2  (Interactive  Search  Designators  Editor).  Program  P2  permits  the  user 
to  edit  the  contents  of  the  three  Range-of-Effect  descriptor  files,  REC,  REV, 
and  REJ.  These  may  have  been  initially  loaded  when  the  Project  Data  Base  was 
generated  using  Program  P 1 7 , however,  if  this  was  not  the  case,  the  entire 
contents  of  these  files  can  be  entered  using  Program  P2.  Figure  III -22  shows  an 
example  edit  of  the  REC  (COURSES)  file.  Here  Program  P2  was  called  by  the  RUNP2 
EXEC  with  the  Project  ID,  PN123,  provided  as  a parameter.  If  Program  P2  were 
entered  from  the  ETAM  EXEC,  the  Project  ID  would  have  been  passed  automatically 
to  the  RUNP2  EXEC. 

The  program  requests  the  user  to  identify  the  file  to  be  edited  by  entering 
either  SELECT  COURSES  (for  the  REC  file),  SELECT  VEHICLES  (for  the  REV  file),  or 
SELECT  TASKS  (for  the  REJ  file).  The  program  then  provides  the  user  with  a 
count  of  the  present  number  of  descriptors  loaded  in  the  file.  When  requested 
to  "MAKE  NEXT  ACTION  REQUEST"  the  user  can  enter  PRINT,  ADD  dn  (dn  = descriptor 
number;  e.g.,  82),  DELETE  dn,  SELECT  another  file  (e.g.,  VEHICLES  or  TASKS),  or 
FILE.  FILE  should  only  be  entered  after  editing  of  all  files  has  been  completed 
since  it  terminates  Program  P2. 

Program  P3  (Range-of-Effect  (ROE)  Search  Module).  Program  P3  is  used  to  search 
the  contents  of  the  Abbreviated  Data  Base  for  entities  which  have  index  descrip- 
tors matching  the  search  descriptors  and  logic  provided  by  the  contents  of  files 
REC,  REV,  and  REJ.  Figure  1 1 1 -23  shows  an  example  of  a run  of  Program  P3.  In 
this  case  it  was  called  by  the  RUNP3  EXEC,  however,  it  can  also  be  selected  as 
one  of  the  options  with  the  Range-of-Effect  portion  of  the  ETAM  EXEC.  No  user 
interaction  is  required  other  than  the  verification  that  the  Project  ID  exists. 

If  no  descriptors  have  been  entered  into  REC,  REV,  or  REJ,  search  of  its  corres- 
ponding Abbreviated  Data  Base  will  be  bypassed.  Program  P3  prints  out  summary 
data  of  the  search  results  as  is  shown  in  Figure  III-23.  Printing  of  the  search 
results  is  accomplished  by  Program  P5A  as  explained  next. 

Program  P5A  (Print  of  Raw  ROE  Results  File).  Program  P5A  prints  the  results  of 
the  Program  P3  search  of  the  Abbreviated  Data  Bases.  Figure  II 1-24  shows  an  ex- 
ample run  of  Program  P5A  as  invoked  by  the  RUNP5A  EXEC.  The  user  has  the  option 
of  directing  the  output  of  this  program  to  either  the  terminal  or  offline  printer, 
or  both.  This  decision  is  aided  by  the  summary  data  provided  by  Program  P3.  If 
there  are  a significant  number  of  entities,  and  the  user  can  wait  for  the  printed 
results  to  be  received  by  mail,  the  output  should  be  directed  to  the  offline 
printer.  As  with  Program  P3,  Program  P5A  provides  a summary  of  the  entities 
processed. 
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css 

llil.45.32  RUNP2  PN123 

PROJECT  NAMED  PM123  ALREADY  EXISTS, 

IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 
S S S 

SEARCH  DESCRIPTOR  EDITOR  RUN  POR  PROJECT  : PM  123 
3-D  PROCEDURAL  TRAINER 


MAKE  INITIAL  ENTR v SELECT 
SELECT  COURSES 


COURSES  SEARCH  DESCRIPTORS  SELECTED 
11  DESCRIPTORS  NOV.'  SELLCTLD 

I m 

::::  MAKE  NEXT  ACTION  REQUEST 
_PP.  I NT 

GROSS  JOB  CATEGORIES 
i,!  OPERATIONS 
31  MAINTENANCE 

OUUECTIVL  I ASK  VARIABLES  AS  MANIFC5T  II.  THE  TRAINING 
4 |I  EQUIPMENT  £ OBJECTS  USED:  REAL 

TASK  FUNCTIONS  DOMINANT  IN  TRAINING 
1.4  procedure  following 

STAGE  OF  LEARNING 
-o.I  ORIENTATION,  FAMILIARIZATION 

-ol  TASK  NOMENCLATURE,  IuLNTS,  LOCATIONS,  CACTS,  RULES 
ij  j PROCEDURES  AT  VERBAL  LEVEL  ONLY 
84  TASK  COMPONENTS  WITH  GUIDANCE 

H5  ENTIRE  JOB-TASK  PROCEDURALLv  AT  BAPELY  ACCEPTABLE  MASTERY 
HU  HIGHLY  PROFICIENT  IN  WORK  CONTEXT 
87  UNUSUAL  TASK  CONDITIONS 

MAKE  I. EXT  ACTION  REQUEST 
DELLTE  87 

— : COURSES  DESIGNATOR  NUMBER  87  HAS  BEEN  DELETED 

MAKi  NL XT  ACTION  REQUEST 
_ADU  82 

ADDED  DESIGNATOR  = 82  TASK  FORMATS  AT  CONCEPTUAL  LEVEL 
:::s  TOTAL  DESIGNATORS  NOW  SEl.CTED  : 11 

::::  MAKE  NEXT  ACT  I fill  REQUEST 
— F I LE 

::::  MODIFIED  CODRStS  DESIGNATORS  HAVE  BEEN  SAVED 
SEARCH  DESIGNATOR  EDITOR  NOW  TERMINATING 


FIGURE  111-22.  EXAMPLE  OF  PROGRAM  P2 
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CSS 

kla.5H.01  RUUP3  PN123 

PROJECT  NAMED  PN123  ALREADY  EXISTS, 

IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 

aSS 

RANGE -OF-EFFECTS  (ROE)  SEARCH  PROGRAM  IS  STARTING 

RANGE-OF-EFFECTS  SEARCH  FOR  PROJECT  : PN123 
3-D  PROCEDURAL  TRAINER 

NOW  PROCESSING  COURSES  SEARCH  - 11  DESCRIPTORS  SELECTED 


TOTAL  NUMUER  DESCRIPTOR  RECORDS  EXAMINED  : 122 

TOTAL  RECORDS  SELECTED  WITH  MATCHING  DATA  ; 37 

TOTAL  RECORDS  SELECTED  WITHOUT  ADORE V I ATED  DATA  : 24 

ACCUMULATED  SEARCH  OUTPUT  RECORDS,  THUS  FAR  : fal 


NOW  PROCESSING  VEHICLES  SEARCH  - (I  DESCRIPTORS  SELECTED 

NO  SEARCH  DESCRIPTORS  SPECIFIED  FOR  VEHICLES  TYPE  DATA  - SEARCH  IS  BYPASSED 
ls::  NOW  PROCESSING  TASKS  SEARCH  - i3  DESCRIPTORS  SELECTED 

NO  SEARCH  DESCRIPTORS  SPECIFIED  FOP.  TASKS  TYPE  DATA  - SEARCH  IS  BYPASSED 

DESIGNATOR  search  program  is  terminating 

TOTAL  ACCUMULATED  SEARCH  OUTPUT  RECORDS  : 01 

FIGURE  111-23.  EXAMPLE  OF  PROGRAM  P3 


1 11-57 


TAEG  REPORT  NO.  40 


CSS 

0 3 . 5 1 . 5 9 RUNP5A  PN123 

PROJECT  NAMED  PNl  2 3 ALREADY  EXISTS, 

IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 


OUTPUT  TO  00  TO  TERMINAL,  OFFLINE  PRINTER,  DOTH,  OR  QUIT? 

ENTER  TERM,  PRTR,  DOTH,  OR  QUIT 

DOTH 

SSLXLCUT ion: 

::::  INITIAL  SEARCH  RESULTS  FOR  PROJECT  -PN123 
3-0  PROCEDURAL  TRAINER 


PACE  NUMBER  : 1 

RANGE -OF-EFFECTS  RESULTS  FOR  : COURSES 


1 

A10200G0 

NOTE 

- NO  MATCHING  NI 

TRAS 

DATA 

AVAILABLE 

PQR 

THIS 

C IN 

2 

A1020093 

NOTE 

- NO  MATCHING  NI 

TRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C If. 

3 

A1O20095 

NOTE 

- NO  MATCHING  NI 

TRAS 

DATA 

AVAILABLE 

FQR 

THIS 

C IN 

9 

A1930050 

2213 

INERT  NAV  PR  INC 

S 

A2920010 

6529 

IS  A 

6 

A9 1 2 0 0 1 0 

6078 

EA-A 

7 

A9 120010 

6287 

EA  A 

8 

A931O01  1 

1035 

EOD  BASIC  NAVY 

9 

A9310019 

1036 

EOD  RE  FR  NAVY 

10 

A9330019 

2090 

MASTER  DIVQUAL 

11 

A9  3 30  (12  5 

2 03  2 

DIVER  FIRST 

12 

A9  910019 

2 09G 

RADIUM  REMOV  OPS 

13 

AS  5 100  1 9 

0133 

CAR  HAND  BAS 

19 

ASS  1002  7 

393  1 

UNREP  MECH/HYj 

IS 

AS  5 10  068 

0139 

FORK  LIFT  OPER 

lb 

AS  800016 

6053 

CTO  A 

17 

Ao  3 1 002  0 

5 399 

1200  PSI  UT 

18 

AO52*050 

5229 

OX  GENR  GL160P 

19 

A b S 2 0 >!  6 8 

9G3E 

SOLAR  GAS  TURBIN 

2 0 

AG7000  1 1 

307  8 

WATCH  REPAIR 

21 

A/010027 

3 3 7M 

WELO/HPRES  PIPE 

22 

A720O0  1 3 

9662 

UT-J 

23 

U3 030051 

NOTE 

- NO  MATCHING  N I 

TRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C IN 

29 

C 003722 

2887 

TA9JF  FAM  PILOTS 

23 

C 003722 

9007 

TA9JF  FAM  PILOTS 

2b 

C 003722 

9009 

TA9JF  FAM  PILOTS 

FIGURE  111-24.  EXAMPLE  OF  PROGRAM  P5A 
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2 7 

C *103722 

9010  TA4JF 

FAM  PILOTS 

28 

C 003722 

9738  TA4JF 

FAM  PILOTS 

29 

C 2C3352 

2570  UH IN  A/C  FAM/P/ 

30 

Cl  003834 

7529  AAS18 

INT  MAI 

- 

31 

Cl  023793 

7914  A 7 AO  ATT  HEAD  I NO 

32 

C1023793 

7915  A7A3  , 

ATT  HEADING 

33 

C1213011 

340 L AWM23 

RADIO  FPEO 

34 

0213011 

544L  AWM23 

RADIO  FREQ 

33 

C6003472 

340C  E2C  WEA  SYS  FAM 

30 

C 0 0 2 3 5 3 0 

34  10  3 4 H b 0 7 7 PROP  INT 

37 

C 640  3 10  3 

7305  CVA/C 

V ALW  SUPV 

38 

D 2C0011 

NOTE  - NO 

MATCHING  NITPAS 

DATA 

AVAILABLE 

FOR 

THIS 

C IN 

39 

L 2A1001 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

this 

C IN 

40 

E 2A1301 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVA ILABLE 

FOR 

THIS 

C IN 

41 

L 2A1803 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C It! 

42 

E 2C0y01 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVA ILABLE 

FOR 

THIS 

C IN 

43 

L 2D001O 

NOTE  - NO 

MATCH  I NO  NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C IN 

44 

E 2JB075 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

TH  I S 

err. 

45 

1.00102  10 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C IN 

40 

L040 104  1 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

TH  I S 

C IN 

4/ 

F7000010 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVA ILABLE 

FOR 

THIS 

C If 

4o 

H 2E3710 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

TH  I S. 

C IN 

49 

J 1 300045 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

c in 

50 

.12210357 

538Q  MULTI 

-THREAT  TNG 

51 

>J  2 3 3020  3 

2037  ADV  EW  OP’S  CRSE 

52 

.12430974 

2 18  1 I PC 

53 

.18800433 

NOTE  - NO 

MATCHING  NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

C 1 rj 

54 

K25300OO 

205 Y SUB  LW  OP- ADV 

55 

L 1 0 1 0 0 2 4 

NOTE  - NO 

MATCHING  NITPAS 

DATA 

AVA ILABLE 

FOR 

TH  I S 

C IN 

PAGE  NUMBER  : 2 

RANGE -OF-E  FFECTS  RESULTS  FOR  : COURSES 


50 

LOO1005O 

NOTE 

- NO 

MATCHING 

NITPAS 

DATA 

AVAILABLE 

FOR 

THIS 

C 

If. 

57 

M198100L 

NOTE 

- NO 

MATCHING 

NITRAS 

DATA 

AVA ILABLE 

FOR 

THIS 

c 

IN 

58 

N 7 0 1 0 3 2 0 

NOTE 

- NO 

MATCHING 

NITRAS 

DATA 

AVAILABLE 

FOR 

THIS 

c 

IN 

59 

Q 2C0015 

NOTE 

- NO 

MATCHING 

NITRAS 

DATA 

AVAILABLE 

FOR 

TH  I S 

c 

IN 

00 

83000010 

NOTE 

- NO 

MATCHING 

NITRAS 

DATA 

AVAILABLE 

POP 

TH  I S 

c 

IN 

01 

83000012 

NOTE 

- NO 

MATCHING 

NITRAS 

DATA 

AVA ILABLE 

FOR 

THIS 

c 

IN 

RESULTS  FILE 

NOW  AT 

END- 

■OF-FILE  : 

;;; 

TOTAL  NUMBER  COURSE  RECORDS  PROCESSED  : 01 

TOTAL  DUMBER  VEHICLE  RECORDS  PROCESSED  : 0 

TOTAL  NUMBER  JOBTASK  RECORDS  PROCESSED  : (I 

TOTAL  NUMBLP.  ROE  RESULT  RECORDS  READ  : Gl 

RECORDS  WITHOUT  MATCHING  ABBREVIATED  DATA  : 24 

TOTAL  NUMBER  DIRECTORY  SEARCH  FAILURES  : 0 


FIGURE  111-24.  EXAMPLE  OF  PROGRAM  P5A  (Cont'd) 
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Program  P5B  (Interactive  Search  Results  Editor).  Program  P5B  permits  the  user 
to  edit  the  contents  of  the  RESU  file  which  contains  the  results  of  the  search 
of  the  Abbreviated  Data  Bases  for  matching  courses,  vehicles,  and  job/tasks. 
Entities  in  the  RESU  file  are  referenced  by  the  sequence  number  with  which  they 
were  associated  in  the  print  out  from  Program  P5B.  If  the  list  is  short,  the 
user  can  print  the  results  of  the  search  by  entering  PRINT  nl  n2  where  nl  is  the 
sequence  number  of  the  first  line  to  be  printed,  and  n2  is  the  sequence  number 
of  the  last  line;  e.g.,  PRINT  1 61  will  print  61  lines.  An  entity  can  be  deleted 
by  entering  DELETE  n where  n is  the  sequence  number  of  the  entity  to  be  deleted. 
Multiple  deletes  may  be  made  by  entering  DELETE  nl  n2  which  will  cause  all 
entities  with  sequence  numbers  from  nl  through  n2  inclusively  to  be  deleted.  An 
entity  is  added  by  entering  ADDC  /entity  id/;  e.g.,  ADDC  /A1 01 01 01 08 , for  courses, 
ADDV  /entity  id/;  e.g.,  ADDV  /691 0lLC003399/ , for  vehicles,  and  ADDJ  /entity 
id/;  e.g.,  ADDJ  /BM/CM/36492/ , for  job/tasks.  Figure  III-26  shows  an  example 
of  a run  of  Program  P5B.  The  results  of  the  editing  are  saved  automatically 
when  the  user  enters  QUIT. 

Program  P5C  (Print  of  ROE  Search  Results  For  Extract).  Program  P5C  permits  the 
user  to  print  a list  of  entities  for  which  additional  data  must  be  obtained 
prior  to  beginning  the  Assessment  provess.  Only  the  original  (non-deleted)  and 
added  entities  will  be  listed  by  this  program.  Figure  1 1 1 -26  shows  an  example 
of  the  print  out  of  Program  P5C.  Output  can  be  directed  to  the  terminal  or 
offline  printer,  or  both.  The  results  of  the  extract  print  are  retained  in  the 
REE  file  of  the  Project  Data  Base  and  can  be  listed  using  Program  PI  (refer  to 
Figure  1 1 1-21  for  an  example  of  a Program  PI  printout  of  the  REE  file  containing 
the  results  of  the  extract  print  shown  in  Figure  1 1 1-26) . 

ETAM  COST  MODELS.  ETAM  is  capable  of  performing  a quantitative  assessment  of 
the  costs/savings  which  will  result  from  introducing  an  innovation.  The  dollar 
effects  are  calculated  by  running  the  appropriate  cost  model  with  the  input 
variables  set  at  some  baseline  state,  then  rerunning  the  model  with  the  parti- 
cular variable  modified  so  that  it  now  reflects  the  value  resulting  from  success- 
ful implementation  of  the  innovation.  Seven  ETAM  programs  are  involved  in  the 
performance  of  this  function.  Three  are  required  to  load  the  extract  files 
(EXC,  EXV,  and  EXJ)  with  model  baseline  input  data.  An  interactive  program 
permits  the  user  to  modify  input  variables  based  upon  the  quantitative  character- 
istics of  the  benefit  pattern  resulting  from  successful  implementation  of  the 
innovation.  There  are  three  model  programs,  i.e.,  training  course,  vehicles, 
and  job/tasks.  The  objectives  in  performing  the  cost  modeling  function  are: 

1.  Load  extract  files  EXC,  EXV,  and  EXJ  with  card  input  data  returned 
from  responsible  Navy  sources.  This  will  essentially  be  a batch 
operation. 

2.  Provide  interactive  capability  to  review,  modify  baseline,  and 
develop  benefit  related  input  variables  for  running  the  cost 
model  programs. 

3.  Select  and  run  the  appropriate  cost  model  program  to  calculate 
the  delta  effect  of  applying  the  innovation  benefits  to  the 
entities  identified  (and  possibly  manually  refined)  during  the 
ETAM  Range-of-Effect  function. 

4.  Store  the  calculated  difference  in  the  CMR  file  for  subsequent 
use  in  the  Decision  Analysis  and/or  Financial  Analysis  routines. 


1 1 1-60 


TAEG  REPORT  NO.  40 


CSS 

Do.5J.5C  RJNP5B  PN123 

PRO.ILCT  NAME  J PN123  ALREADY  EXISTS, 

IS  THIS  CORRECT?  CRESPOND  YES  OR  QUIT) 

YES 
5 5 55 

ROE  SEARCH  RESULTS  EDIT  TOR  PROJECT  PH  123 
3-D  PROCEDURAL  TRAINER 


so:  (,  2 HAVE  BEEN  LOADED  INTO  MEMORY  TOR  EDIT  INC 

::::  MARE  NEXT  ACTION  REDDEST 
DELETE  1 3 

DELETED  RECORD  NUMBER  : 1 THRU  3 

■ss:  MAKE  NEXT  ACTION  REOUEST 
_DE  LETE  23 

DELETED  RECORD  NUMBER  : 23 

sss:  MAKE  NEXT  ACTION  REOUEST 
_DELETE  38  49 

dlletld  RECORD  NUMBER  : 38  THRU  49 

so*  MAKE  NEXT  ACT  ION  REOUEST 
..DELETE  53 

DELETED  RECORD  NUMBER  : 53 

sss:  MAKE  NEXT  ACT  ION  REOUEST 
DELETE  55 

deleted  record  number  : 55 

sss:  MAKE  NEXT  ACTION  REOUEST 
_ADDC  /A1010108/ 

THE  FOLLOWING  RECORuCS)  HAS/HAVE  BEEN  ADDED... 

RECNO/C I N/CDP/ COURSE  : 6 2/ A 1 0 1 0 1 0 C 7 6 5 4 / WRA-4  C MB  MA 

sss:  MAKE  NEXT  ACTION  REOUEST 
_PR  I NT  1 Cl 

RECNO/C IN/CDP/T  ITLE  : 1/A1020060/  / CNO  MATCHING  DATA) 

RECNO/C IN /CUP/TITLE  : 2/A1020093/  / CNO  MATCHING  DATA) 

RECNO/C IN/COP/TITLE  I 3/A1020095/  / (NO  MATCHING  DATA) 

RECNO/C IN/CDP/T ITLE  : 4 / A 1 9 30 0 5 0 / 2 2 1 3 / INEPT  NAV  PRINC 

RECNO/CIN/CDP/TITLL  : 5 / A 2 4 20 0 1 0 / C, 5 2 9 / IS  A 


RECNO/C  IN/CDP/T ITLE  : C /A4  1 2 0 0 1 0 / C 0 7 8 / EA-A 


FIGURE  111-25.  EXAMPLE  OF  PROGRAM  P5B 
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RECNO/C  IN/CDP/T  ITLE  : 
RECNO/CIN/CDP/TITLE  : 
RECNO/C IN/CUP/T  ITLE  : 
RECNO/CIN/CDP/TITLE  : 
RLCNO/C  IN/CDP/T  ITLE  : 
RECNO/C  IN/CUP/T  ITLt  : 
RLCNO/C  IN/CUP/T  ITLE  : 
RLCNO/C  IN/CUP/T  ITLE  : 
KLCUO/C  IN/CUP/T  ITLE  : 
RLCNO/C  IN/CUP/T ITLE  : 
RECNO/C  IN/CUP/T ITLL  : 
RECNO/C  IN/CDP/T ITLE  : 
RLCNO/C  IN/CUP/T  ITLE  : 
RECNO/C  IN/CUP/T ITLE  : 
RLCNO/CIN /CUP/TITLE  : 
RECfJO/C  IN/CUP/T  ITLE  : 
RECNO/C  IN/CDP/T ITLE  : 
RECNO/C IN/CUP/T ITLE  : 
RLCNO/C  IN/CUP/T ITLE  : 
RLCNO/CIN /CUP/T ITLE  : 
RECNO/C  IN/CUP/T ITLE  : 
RLCNO/C IN/CUP/T ITLE  : 
RECNO/C  IN/CDP/T ITLE  : 
RECNO/C IN/CUP/T  ITLE  : 
RECNO/C IN/CUP/T ITLE  : 
RLCNO/C  IN/CDP/T ITLE  : 
RECNO/C IN/CUP/T ITLE  : 
RLCNO/C  IN/CUP/T  ITLE  : 
RECNO/C IN/CUP/T  ITLE  : 

FIGURE  lli-25. 


7/A4  120010/6287/ 
8/A431 0011/1035/ 
9/A4310014/1036/ 
10/A4330019/2040/ 
11/ A4 33002 5/2 082/ 
1 2 / A4  9 10014/2040/ 
13/A551 0019/0133/ 
14/A5510027/3931/ 
15/A5 5 10068/0134/ 
16/A5800016/6053/ 
17/ AC  51002 0/5399/ 
18/A6520050/5224/ 
19/AG5200L8/4U3E/ 
20/AG70001 1/3078/ 
21/A7010027/337M/ 
22/A7200013/4GGZ/ 
23/03030051/  / 
24/C  003722/2887/ 
25/C  003722/9007/ 
2G/C  003722/9009/ 
27 /C  003722/9010/ 
20/C  003722/9738/ 
2 9/c  2C  3352/2576/ 
30/C  1003834/7529/ 
31/C1023793/7914/ 
32/C  102  3793/  79  15/ 
33/C12130  11/34  tiL/ 
34/C1213011/544L/ 
35/CG003472/340C/ 


EA  A 

EOD  BASIC  NAVY 
EOD  REFR  NAVY 
MASTEP  DIVOUAL 
UIVCR  F I 0 ST 
RADIUM  RE  MOV  OPS 
CAR  HAND  BAS 
UN REP  MECH/HYD 
FORK.  LIFT  OPER 
CTO  A 

1200  PSI  BT 
OX  GENR  GL1G0P 
SOLAR  GAS  TURBIN 
WATCH  REPAIR 
WELD/HPRES  PIPE 

UT-il 

C NO  MATCHING  DATA) 
TA4UF  FAM  PILOTS 
TA4JF  FAM  PILOTS 
TA4JF  FAM  PILOTS 
TA4JF  FAM  PILOTS 
TA4.IF  FAM  PILOTS 
UH1N  A/C  FAM/P/ 
AAS18  I NT  MAI 
A 7 AB  ATT  HEADING 
A 7 A B ATT  HEADING 
AWM23  RADIO  FPEO 
AWM23  RADIO  FREO 
E2C  WEA  SYS  TAM 


EXAMPLE  OF  PROGRAM  PSB  (Cont'd) 
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RLCNO/C IN/CDP/TITLE  : 

36/C6023536/3O  ID/ 

5 4H6 0 7 7 PROP 

INT 

RLCNO/C IN/CDP/T ITLE  : 

37/C6463103/73G5/ 

CVA/CV  ALU  SlIPU 

RLCNO/C IN/CDP/T ITLE  : 

38/D  2C0011/ 

/ 

(NO 

MATCHING 

DATA) 

RLCNO/C IN/CDP/T ITLE  : 

30/E  2A1001/ 

/ 

CNO 

MATCH  INC. 

DATA) 

RLCNO/C IN/CDP/T ITLE  : 

40/E  2A 1 30  1 / 

/ 

(NO 

MATCHING 

DATA) 

RECNO/C IN/CDP/T  ITLE  : 

41/E  2A180  3/ 

/ 

(NO 

MATCHING, 

DATA) 

RLCNO/C IN/CDP/T ITLE  : 

42/E  2C090  1/ 

/ 

(NO 

MATCHING 

DATA) 

RLCNO/C  IN/CDP/TITLE  : 

43/E  200016/ 

/ 

(NO 

MATCHING 

DATA) 

RLCNO/C  IN/CDP/T ITLE  : 

44/E  2D0075/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/C IN/CDP/T ITLE  : 

45/E60  10210/ 

/ 

CNO 

MATCHING 

DATA) 

RECNO/C IN/CDP/T ITLE  : 

4G/L6461G41/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/C IN/CDP/T  ITLE  : 

‘♦7/F7000010/ 

/ 

CNO 

MATCHING, 

DATA) 

RLCNO/C IN/CDP/T ITLE  : 

4o/H  2E3710/ 

/ 

CNO 

MATCHING 

DATA) 

rlcno/cin/cdp/title  : 

44/01300645/ 

/ 

CNO 

MAT  CHINO 

DATA) 

RECNO/C  IN/CDP/T  ITLE  : 

50/0 2210357/538 Q/ 

MULT  I -THREAT 

TNG 

rlcno/cin/cdp/title  : 

51/ 023 30 203/2 G 37/ 

ADV 

EW  OP'S  CRSE 

RLCNO/CIN/CDP/TITLE  : 

52/J2430O7O/2 18 1/ 

I PC 

RLCNO/CIN/CDP/TITLE  : 

53/08000433/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/CIN/CDP/TITLE  : 

5R/K2330066/205Y/ 

SUB 

EW  OP-ADV 

RECNO/C IN/CDP/T ITLE  : 

55/Ll  0 10020/ 

/ 

CNO 

MATCHING 

DATA) 

RECNO/C IN/CDP/T ITLE  : 

56/L6610056/ 

/ 

CNO 

MATCH  INC 

DATA) 

RLCNO/CIN/CDP/TITLE  : 

57/Ml  0 81 00L/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/CIN/CDP/TITLE  : 

58/ N 701 0320/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/CIN/CDP/TITLE  : 

59/Q  2C0  01 5 / 

/ 

CNO 

MATCH ING 

DATA) 

rlcno/cin/cdp/title  : 

60/83000010/ 

/ 

CNO 

MATCHING 

DATA) 

RLCNO/CIN/CDP/TITLE  : 

61/83000012/ 

/ 

CNO 

MATCHING 

DATA) 

:t::  MAKE  NEXT  ACTION  REOUEST 
_QUIT 

::::  ROE  SEARCH  RESULTS  EDITOR  IS  TERM  I NAT  INC 
TOTAL  RECORDS  ADDED  : 1 

TOTAL  RECORDS  IN  PROJECT  tXTRACT  FILE  : 44 

FIGURE  111-25.  EXAMPLE  OF  PROGRAM  P68  (Cont'd) 
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CSS 

1J9.13.5H  RUNP5C  PN123 

PROJECT  NAMED  PN123  ALREADY  EXISTS, 

IS  THIS  CORRECT?  (RESPOND  YES  OR  QUIT) 

YES 


OUTPUT  TO  GO  TO  TERMINAL,  OFFLINE  PRINTER,  DOTH,  OR  QUIT? 

ENTER  TERM,  PRTR,  BOTH,  OR  QUIT 

BOTH 

S$0  0 PN123 


KEYWORD  OR  ARGUMENT  ERROR 
EXLCUT ion: 

::::  EXTRACT  SEARCH  RESULTS  FOR  PROJECT  - PN123 
3-D  PROCEDURAL  TRAINER 


PAGE  NUMBER  : 1 


1 

ORIGINAL 

A1930050 

2213 

INERT  NAV  PR  INC 

2 

ORIGINAL 

A24200  10 

6529 

IS  A 

3 

ORIGINAL 

A4 120010 

6078 

EA-A 

4 

ORIGINAL 

A4 120010 

6287 

EA  A 

5 

ORIGINAL 

A4310011 

1035 

EOD  BASIC  NAVY 

0 

ORIGINAL 

A4310014 

1036 

EOD  RE  FR  NAVY 

7 

ORIGINAL 

A4  3 300  1 9 

2040 

MASTER  DIVOIJAL 

8 

ORIGINAL 

A4  3 3002  5 

2082 

DIVER  FIRST 

9 

ORIGINAL 

A4910014 

204G 

RADIUM  REMOV  OPS 

10 

ORIGINAL 

A55 10019 

0133 

CAR  HAND  BAS 

11 

ORIGINAL 

A5510027 

393  1 

UNREP  MECH/HYD 

12 

ORIGINAL 

A5510068 

0134 

FORK  LIFT  OPER 

13 

ORIGINAL 

A5800016 

0053 

CTO  A 

14 

ORIGINAL 

AO  5 1002  0 

5399 

1200  PSI  BT 

15 

ORIGINAL 

AO520050 

5224 

OX  GENR  6L160P 

10 

ORIGINAL 

Ab520068 

463E 

SOLAR  GAS  TURBIN 

17 

ORIGINAL 

AO  70001  1 

3078 

WATCH  REPAIR 

18 

ORIGINAL 

A7010027 

33  7 M 

WELD/HPRES  PIPE 

19 

ORIGINAL 

A7  2 000  1 3 

406Z 

UT-J 

FIGURE  111-26.  EXAMPLE  OF  PROGRAM  P5C 
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20 

ORIGINAL 

C 003722 

2887 

TA4JF  FAM  PILOTS 

21 

ORIGINAL 

C 003722 

9007 
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These  objectives  are  accomplished  through  seven  programs,  namely: 

P10  - Course  Extract  Data  Base  Load 
Pll  - Vehicle  Extract  Data  Base  Load 
PI 2 - Job/Task  Extract  Data  Base  Load 
P51  - Interactive  Benefit  Pattern  Setup 
P52  - Training  Course  Cost  Model 
P53  - Vehicles  Cost  Model 
P54  - Job  Task  Cost  Model 

These  programs  are  supported  by  four  Project  Data  Base  files:  EXC,  EXV,  and 

EXJ  which  store  the  extract  input  data,  CMR  which  stores  the  models'  output 
data,  and  two  temporary  files  which  are  set  up  with  the  baseline  state  and 
the  benefit  state  of  the  cost  models'  input  variables. 

The  programs  and  data  files  to  support  the  ETAM  Cost  Model  part  of  the  ETAM 
Assessment  function  are  outlined  in  this  section.  Figure  1 1 1-27  shows  the 
interrelationships  between  the  Programs  P51  through  P54  and  Executive  Control. 

Program  PI 0 (Course  Extract  Data  Base  Load). 

Purpose:  This  program  loads  the  EXC  file  from  cards  which  contain  the  addi- 

tional data  on  Courses  required  for  the  ETAM  Assessment  function. 

Inputs:  Refer  to  Figure  1 1 1-6  EXC  DATA  ELEMENTS  for  a list  of  the  input 

variables,  their  record  length,  and  format.  The  input  cards  will 
be  formatted  as  follows: 

CARD  NO.  VARIABLE  FROM/THROUGH  REC  POS  CARD  COLS 


1 

CDP  . . . 

PESP 

1-79 

1-79 

2 

TLEGTH  . 

. . DRATE 

80-157 

1-78 

3 

INRATE  . , 

. . GRAD (11 ) 

158-234 

1-77 

4 

GRAD  (12) 

. . . C0PMT(5) 

235-313 

1-79 

5 

C0PMT(6) 

. . . C0PMT(20) 

314-388 

1-75 

Missing  data  should  be  unpunched  (not  zeros).  Cards  will  be  sequence 

numbered  1-5  in  Col.  80. 

Process:  The  major  program  process  steps  are: 

1 . Read  input  cards. 

2.  Check  sequence  numbers  for  order  and  for  missing  cards.  If 
the  variable  N (Planning  Period  (Years))  is  5 or  less,  then 
Card  No.  5 is  not  required,  (although  this  is  optional)  since 
no  data  would  exist  for  C0PMT(6)  through  C0PMT(20). 

3.  Print  appropriate  error  messages. 

4.  List  data  formatted  with  variable  headings  for  readability 

by  the  user.  Provide  the  list  option  for  terminal  or  offline 
printer,  or  both. 

5.  Identify  missing  data  (unpunched)  in  the  listing  with  an  asterisk 

(*). 

6.  Create  single  388  position  record  to  be  written  in  the  EXC  file. 

7.  Compare  returned  data  with  Entity  Type  1 records  in  the  REE  file 
for  matching  CDP's. 

8.  Print  list  of  unmatched  CDP's  in  either  file.  This  list  should 
be  directed  to  the  terminal. 
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Output:  The  outputs  of  Program  PI 0 will  be: 

1.  A 388  position  record  formatted  as  outlined  in  Figure  1 1 1 -6  EXC 
DATA  ELEMENTS  to  be  written  to  the  EXC  file  (one  record  per  each 
sequence  of  5 (or  4)  input  cards. 

2.  A report  listing  the  variables  for  each  CDP  for  which  data 
have  been  provided. 

3.  A report  identifying  unmatched  CDP's  with  the  original  extract 
listing  from  Program  P5C,  (these  are  contained  in  the  REE  file). 

Program  PI  1 (Vehicle  Extract  Data  Base  Load). 

Purpose:  This  program  loads  the  EXV  file  from  cards  which  contain  the  addi- 

tional data  on  Vehicles  required  for  the  ETAM  Assessment  function. 

Inputs:  Refer  to  Figure  1 1 1 -7  EXV  DATA  ELEMENTS  for  a list  of  the  input 

variables,  their  record  length,  and  format.  The  input  cards  will 
be  formatted  as  follows: 


CARD  NO.  VARIABLE  FROM/THROUGH  REC  POS  CARD  COLS 


1 VFSN  . . . PEQU I P ( 6 ) 1-76  1-76 

2 P EQU I P ( 7 ) . . . PEQUIP(IS)  77-1  54  1-78 

3 PEQU I P ( 20 ) . . . COPMT (14)  155-230  1-76 

4 COPMT (15)  . . . C0PMT(20)  231-260  1-30 


Missing  data  should  be  unpunched  (not  zeros).  Cards  will  be  sequence 
numbered  1-4  in  Col.  80. 

Process:  The  major  program  process  steps  are: 

1 . Read  input  cards. 

2.  Check  sequence  numbers  for  order  and  for  missing  cards.  All 
four  cards  are  required  for  each  Vehicle  Entity. 

3.  Print  appropriate  error  messages. 

4.  List  data  formatted  with  variable  headings  for  readability 

by  the  user.  Provide  the  list  option  for  terminal  or  offline 
printer,  or  both. 

5.  Identify  missing  data  (unpunched)  in  the  listing  with  an  asterisk 
(*)• 

6.  Create  single  260  position  record  to  be  written  in  the  EXV  file. 

7.  Compare  returned  data  with  Entity  Type  2 records  in  the  REE  file 
for  matching  Vehicle  Federal  Stock  Numbers. 

8.  Print  list  of  unmatched  Vehicle  Federal  Stock  Numbers  in  either 
file.  This  list  should  be  directed  to  the  terminal. 

Output:  The  outputs  of  Program  Pll  will  De: 

1.  A 260  position  record  formatted  as  outlined  in  Figure  1 1 1 -7  EXV 
DATA  ELEMENTS  to  be  written  to  the  EXV  file  (one  record  per  each 
sequence  of  4 input  cards). 
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2.  A report  listing  the  variables  for  each  Vehicle  Federal  Stock 
Number  for  which  data  have  been  provided. 

3.  A report  identifying  unmatched  Vehicle  Federal  Stock  Numbers 
with  the  original  extract  listing  from  Program  P5C,  (these  are 
contained  in  the  REE  file.). 

Program  PI 2 (Job/Task  Extract  Data  Base  Load). 

Purpose:  This  program  loads  the  EXJ  file  from  cards  which  contain  the  addi- 

tional data  on  Job/Tasks  required  for  the  ETAM  Assessment  function. 

Inputs:  Refer  to  Figure  1 11-8  EXJ  DATA  ELEMENTS  for  a list  of  the  input 

variables,  their  record  length,  and  format.  The  input  cards  will 
be  formatted  as  follows: 


CARD  NO. 

VARIABLE  FROM/THROUGH 

REC  POS 

CARD  COLS 

1 

JTID  . . . 

INRATE 

1-77 

1-77 

2 

APERS(l)  . 

. . APERS( 1 3 ) 

78-155 

1-78 

3 

APERS( 14)  . 

. . APERS(20) 

156-197 

1-42 

Missing  data  should 
numbered  1-3  in  Col . 

be  unpunched 
. 80. 

(not  zeros) . 

Cards  will 

be  sequence 

Process:  The  major  program  process  steps  are: 

1 . Read  input  cards. 

2.  Check  sequence  numbers  for  order  and  for  missing  cards.  If 
the  variable  N (Planning  Period  (Years))  is  13  or  less,  then 
Card  No.  3 is  not  required,  (although  this  is  optional)  since 
no  data  would  exist  for  APERS(14)  through  APERS(20). 

3.  Print  appropriate  error  messages. 

4.  List  data  formatted  with  variable  headings  for  readability 

by  the  user.  Provide  the  list  option  for  terminal  or  offline 
printer,  or  both. 

5.  Identify  missing  data  (unpunched)  in  the  listing  with  an  asterisk 
(*)• 

6.  Create  single  197  position  record  to  be  written  in  the  EXJ  file. 

7.  Compare  returned  data  with  Entity  Type  3 records  in  the  REE  file 
for  matching  Job/Task  Identifiers. 

8.  Print  list  of  unmatched  Job/Task  Identifiers  in  either  file. 

This  list  should  be  directed  to  the  terminal. 

Output:  The  outputs  of  Program  P12  will  be: 

1.  A 197  position  record  formatted  as  outlined  in  Figure  1 1 1 -8  EXJ 
DATA  ELEMENTS  to  be  written  to  the  EXJ  file  (one  record  per  each 
sequence  of  3 (or  2)  input  cards. 

2.  A report  listing  the  variables  for  each  Job/Task  Identifier  for 
which  data  have  been  provided. 

3.  A report  identifying  unmatched  Job/Task  Identifiers  with  the 
original  extract  listing  from  Program  P5C , (these  are  contained 
in  the  REE  file). 
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Program  P51  (Interactive  Benefit  Pattern  Setup). 

Purpose:  The  objective  of  Program  P51  is  to  establish  interactively  the  pattern 

of  benefits  which  will  be  accruing  from  the  innovation  reflected  in  a 
modification  of  the  variables  which  are  inputs  to  the  three  cost 
models.  The  baseline  values  for  the  cost  models  input  variables 
are  stored  in  the  Project  Data  Base  files  EXC,  EXV,  and  EXJ.  In 
general,  these  values  can  be  used  directly  and  moved  to  the  CMIA 
file  for  input  to  the  baseline  (first  pass)  run  of  the  cost  model's 
processing  sequence.  However,  the  user  has  the  option  of  modifying 
any  of  the  variables  stored  in  the  three  extract  files  prior 
to  moving  them  to  the  CMIA  file.  The  user  then  is  permitted  to  apply 
the  benefit  pattern  as  reflected  descriptively  in  the  Project  Data 
Base  file  BPT  to  the  input  variables  prior  to  moving  them  to  the  CMIB 
file.  The  benefit  pattern  is  established  fo,  any  one  or  combination 
of  input  variables  by  indicating  a new  value  for  that  variable  or 
some  percentage  change  to  be  applied  to  that  variable. 

Inputs:  The  inputs  to  Program  P51  are  the  contents  of  the  EXC,  EXV,  and  EXJ 

files,  which  were  previously  shown  in  Figures  1 1 1 -6 , 7 and  8.  The  user 
provides  control  input  via  the  terminal  by  selecting  one  of  the  available 
options  provided  by  Program  P51 . 

Process:  Following  are  the  major  process  steps  for  Program  P51 : 

1.  The  user  accesses  this  program  by  entering  RUNP51  EXEC  or  by 
the  select  option  within  the  ETAM  EXEC  called  MOD. 

2.  The  user  is  then  presented  with  a display  menu  of  the  various 
options  that  he  can  select  at  this  time.  This  menu  will  be  as 
follows: 

o Review  extract  files 

Under  this  option  the  user  will  be  allowed  to  review  the 
contents  of  any  of  the  extract  files  EXC,  EXV,  or  EXJ. 
o Review  BPT  file 

Normally  the  user  would  refer  to  a listing  of  the  Project 
Data  Base  previously  printed  from  Program  PI.  However, 
if  the  user  wishes  to  scan  the  BPT  file  at  this  time,  he 
will  have  access  to  it  through  this  select  option, 
o Establish  CMIA  file 

Here  the  user  will  be  allowed  to  either  transfer  the  contents 
of  the  extract  files  directly  to  CMIA  or  permitted  to  make 
modifications  if  new  data  exists  at  this  time.  The  modifi- 
cation of  extract  data  going  to  the  CMIA  file  will  generally 
be  on  a straight  value  change  basis  rather  than  a percentage 
change  as  will  be  possible  under  the  benefit  modification  of 
data  going  to  the  CMIB  file, 
o Review/print  the  CMIA  file 

The  user  will  be  allowed  to  print  the  contents  of  the  CMIA 
file  under  this  select  option, 
o Establish  CMIB  file 

The  CMIB  file  will  contain  the  variables  as  modified  to 
reflect  the  benefit  pattern  accruing  from  the  innovation. 

At  this  point  the  user  will  select  the  specific  variable  or 
variables  from  the  display  of  the  EXC,  EXV,  or  EXJ  files  and 
make  the  appropriate  modification,  either  by  a value  change 
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or  by  indicating  a percentage  change  from  its  baseline  value. 

The  result  of  this  modification  after  all  variables  have  been 
treated,  will  be  moved  to  the  CMIB  file  as  input  to  the  second 
pass  of  the  Cost  Model  runs, 
o Review  CMIB  file 

Under  this  option  the  user  will  have  two  possible  selections. 
First,  the  CMIB  file  can  be  reviewed  as  was  done  with  the 
CMIA  file.  Second,  the  user  can  request  only  those  variables 
which  differ  from  the  CMIA  file  to  be  printed.  Under  this 
option  both  the  original  value  in  the  CMIA  file  and  the  new 
value  in  the  CMIB  file  will  be  printed, 
o Run  Cost  Model 

' Any  one  or  all  of  the  Cost  Models  can  be  selected  to  run  against 
the  input  variables  in  the  CMIA  and  CMIB  files  at  this  time. 
Appropriate  error  messages  will  be  indicated  if  the  CMIA  and 
CMIB  files  have  not  previously  been  set  up.  Also,  as  a part 
of  this  selection  step,  the  user  can  select  any  of  the  report 
options  available  with  each  of  the  cost  models.  For  example, 
under  the  training  cost  model,  seven  report  outputs  are  select- 
able. The  user  specifies  which  of  these  reports  are  desired 
to  be  printed  and  also,  the  user  will  direct  the  output  to 
either  the  terminal  or  the  offline  printer,  or  both, 
o Quit 

Under  this  option  the  user  returns  to  the  next  higher  level 
of  EXEC,  which  in  this  case  would  permit  the  selection  of 
another  option  under  Assessment. 

3.  Following  completion  of  any  one  of  the  options  selected  under  Step 
2,  the  user  will  be  returned  to  the  display  permitting  another  option 
to  be  selected. 


Outputs:  The  major  outputs  from  the  benefit  pattern  setup  program  are: 

1.  A series  of  display  prompts  which  let  the  user  select,  via  the 
terminal,  the  various  options  under  this  program. 

2.  A variety  of  reports  which  display  the  results  of  selecting  any 
one  of  the  options  identified  under  Step  2.  For  example,  if  a 
review  of  the  BPT  file  is  selected,  an  appropriate  report  will 
be  printed.  In  this  case,  the  report  would  be  the  same  as  that 
generated  under  Program  PI.  All  reports  generated  from  this 
program  will  be  formatted  with  appropriate  headings  for  user 
readability.  The  user  will  also  have  the  option  of  selecting 
either  the  terminal  or  offline  printer,  or  both. 

3.  Although  not  a direct  output  of  Program  P51,  selection  of 
the  Cost  Models  to  be  run  will  cause  certain  reports  to  be 
generated  from  the  respective  Cost  Model  programs.  The 
selection  of  these  reports  is  under  the  control  of  Program 
P51;  i.e.,  Program  P51  sets  up  the  run  parameters  for  the  Cost 
Models  identifying  the  reports  to  be  printed. 


Program  P52  (Training  Cost  Model 


Purpose:  The  primary  objective  of  ETAM  is  to  evaluate  the  effects  of  an  innova- 

tion which  has  been  implemented  within  the  training  system.  In  order 
to  perform  this  function,  a model  capable  of  calculating  training  costs 
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under  different  input  assumptions  was  required.  A model  had  been  de- 
veloped for  TECEP  (Training  Effectiveness,  Cost  Effectiveness  Prediction',* 
technique  which  had  most  of  the  features  required  for  performing  the 
training  calculations.  The  basic  logic  of  the  program  has  been  incor- 
porated into  the  Training  Cost  Model  outlined  here.  A program  list- 
ing has  been  included  in  Appendix  C of  this  report.  Several  features 
which  were  added  to  the  original  program  are: 

1)  Extensive  input  data  editing  and  default  data  specification. 

2)  User  selectable  policies  for  equipment  purchases  and  equip- 
ment depreciation. 

3)  Selectable  output  reports. 

4)  Output  format  design  is  compatible  with  80X24  CRT  displays 

to  facilitate  any  conversion  fnam  the  present  thermal  printer 
to  a CRT  display  capability. 

Extensive  commenting  within  the  program  was  done  to  guide  the  pro- 
grammer in  the  final  incorporation  of  the  Training  Cost  Module 
into  the  ETAM  Training  Cost  Model.  The  model  has  been  programmed 
to  sequence  through  the  calculations  twice,  once  to  compute  the 
baseline  technical  and  cost  factors  and  another  time  to  compute 
these  factors  with  the  benefits  pattern  developed  in  Program  P51 
applied.  The  program  output  intended  for  the  CMR  file  is  two  twenty 
year  vectors  providing  annual  investment  costs  and  annual  recurring 
cost/savings.  Examples  of  the  model  outputs  are  shown  later  in  this 
section. 

Input:  The  data  input  variables  required  to  run  the  Training  Cost  Model  were 

set  up  by  Program  P51 . The  baseline  input  variables  are  contained 
in  the  Cost  Model’s  inputs  W/0  Benefits  (CMIA)  file  and  would  generally 
reflect  the  EXC  file  unless  modified  by  Program  P51 . The  input  variables 
with  benefits  applied  as  set  up  by  Program  P51  are  contained  in  the 
Cost  Model's  input  With  Benefits  ( CM  IB)  file. 

The  input  variables  are  the  same  as  described  in  Figure  1 1 1 -6 , EXC 
DATA  ELEMENTS.  Since  the  program  design  with  the  descriptions  of 
input  variables  is  explained  in  the  previously  referenced  report  on 
TECEP,  only  the  input  variables  which  have  been  modified  from  the 
original  model  are  explained  here. 

1)  Equipment  Purchase  Policies  (PURCHF) 

The  Training  Cost  Model  permits  the  user  to  specify  any  one 
of  four  purchase  policies  with  regards  to  the  equipment 
calculations  within  the  Training  Cost  Model. 

Policy  1 (Input  Variable  PURCHF  = 1)  causes  the  total 
average  requirements  for  equipment  to  be  purchased  in  the 
first  year  of  the  equipment  life.  Under  this  purchase 
option  a buildup  of  training  in  which  lesser  equipment  re- 
quirements exist  in  the  earlier  years  would  not  be  taken 
into  consideration. 


* Braby,  Richard,  Ed. D;  Henry,  James  M.;  Parrish,  William  F.,  Jr.;  Swope, 

William  M.,  Ph.D. , 1975.  A Technique  for  Choosing  Cost-Effective  Instructional 
Delivery  Systems.  TAEG  Report  No.  16,  Training  Analysis  and  Evaluation  Group, 
Orlando,  FL. 
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Policy  2 (input  variable  PURCHF  = 2)  causes  the  total  maxi- 
mum requirement  for  equipment  over  the  equipment  life  to  be 
purchased  in  the  first  year.  This  is  similar  to  Policy  1, 
however,  in  Policy  2 the  maximum  requirement  rather  than  the 
average  requirement  for  equipment  is  purchased. 

Policy  3 (input  variable  PURCHF  = 3)  causes  equipment  to  be 
purchased  up  to  the  average  requirement  but  with  each  purchase 
the  requirement  within  the  specific  year  of  purchase  is  not 
exceeded.  This  policy  provides  a more  efficient  expenditure 
of  resources  for  equipment  in  relationship  to  a buildup  of 
training  loaas  over  the  planning  period. 

Policy  4 (input  variable  PURCHF  = 4)  causes  equipment  to  be 
purchased  up  to  the  maximum  requirement  over  the  equipment 
life  but  not  to  exceed  the  requirements  in  the  specific  year 
of  purchase.  This  policy  is  similar  to  Policy  3,  however, 
the  maximum  rather  than  average  numbers  of  equipment  are  pur- 
chased. 

2.  Depreciation  Policies  (DEPF) 

Policy  1 (input  variable  DEPF  = 1)  causes  equipment  to  be  de- 
preciated linearly  over  its  life. 

Policy  2 (input  variable  DEPF  = 2)  causes  a sum  of  the  digits 
depreciation  to  take  place  over  the  equipment  life. 

Policy  3 (input  variable  DEPF  = 3)  causes  the  equipment  to  be 
fully  depreciated  over  its  life.  Equipment  purchased  under  this 
policy  would  have  no  remaining  value  at  the  end  of  the  planning 

period. 

3.  Percent  of  Maximum  Equipment  Purchased  (PCTPCH) 

The  user  can  specify  the  percent  of  calculated  equipment 
requirements  which  are  to  be  purchased  under  the  specific 
purchase  policy  selected.  This,  in  effect,  permits  the 
maximum  requirement  for  equipment  calculated  by  the  model 
to  be  limited. 

4.  Inflation  Rate  (INRATE) 

The  user  can  have  cost  calculations  made  in  four  modes  using 
the  training  cost  model.  The  first  is  a non-di scounted  cost 
per  year,  the  second  is  a discounted  cost  per  year  based  upon 
the  discount  rate  specified,  the  third  is  an  inflated,  non- 
discounted  cost  per  year  calculation  which  gives  costs  n 
current  dollars,  and  fourth  is  the  stream  of  costs,  previously 
inflated,  discounted  at  the  discount  rate  specified. 

Process:  The  major  program  process  steps,  within  the  Training  Cost  Model  are: 

1.  Setup  model  to  sequence  through  calculations  using  input  from  the 
CM  I A file. 
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2.  Read  input  variables  from  first  Entity  Type  1 (courses)  record. 

3.  Calculate: 

o Technical  Processing  Logic 


INPUT 

LOGIC 

DESCRIPTION 

OLTP  * 

GRA3( I ) 

Input  graduates  required  annual*. 

'/IRATE 

STUO(I)  « GRAD(I)/(1 .0-ARATE) 

N 

Calculate  student  input  rqmts  'row 
annual  graduates  required  and 
attrition  rate. 

STUD( I ) 

N 

AASIN  • ( Z STUD(I))/N 
1-1 

Sunsnarlre  average  "unGer  of  student 
1 nputs  required  over  planning 
period  N . 

AASIA 

TLENGH 

STUMf(I  ) ■ (TLENGH*$TUQ(  I ) )•(  1 .0- . S*ARAT  E ) 

Calculate  student  manweeks  'rom 
noma ' flow  tnrough  course  including 

ettrites. 

STlDNN'I. 

RCRATE 

ARCTT* 

SMURRC ( I ) • RCRATE*STUO(I)*ARCVTM 

Calculate  student  manweeks  from  set- 
back students  recycled. 

MSCHOP 

AOB(I)  ■ (STUDMWd )*SHWPRC(I))/WSCHOP 

Calculate  eve rage- on- board  from 
student  manweeks  distributed  across 
school  operating  period. 

AOB(l) 

AA08  « (C  AOB(  X ))/N 

M 

Smwnarlze  the  average  average-on- 
board over  planning  period  N. 

AA0f> 

TSPOSO 

PESP 

PSP ( I } • ( ( ACB(  I ' T-TSPCSD))*(1  FUPESP) 

Calculate  annual  student  position 

PSP(I) 

WSMOP1 

(HSCHOP/wSHon i 

requirements  considering  average 
unavailability  and  a percentage 
e*t re  positions  for  Queueing. 

INTSPO 

RINSTR(I)  ■ PSPil >*INTSPC 

Caculate  annual  instructor  rqmts 
using  an  Instructor  to  student 
position  ratio 

CINSTR(I) 

AHTSPG 

RADMIN(I)  • PSP(!)*AKTSPO 

Calculate  annual  adulnl strati ve 
requirements  using  an  admin,  to 
student  position  ratio 

RACWINd' 

EQTSPO 

REQUIP(l)  * PSP(1)*EQTSP0 

Calculate  annual  equipment  rqmts 
using  an  equip  to  student 
position  ratio 

REoup(r; 

PURCHF 

LOFEQ 

PEOUIPl)  * f(puPChf'  PfO"IP(! l.LOPEO) 
1 J 

Calculate  actual  nuefcer*  of  equip- 
ment purchased  annually  based  on 
selected  purchase  policy  and  equip- 
ment life. 

PEOUIPl I 

aeouip(j)  ■ Z 2I?equip  : 
J*N  i«j-lofeo*' 

Calculate  available  eouiDment  on 
board  annually  over  the  plannmq 
period 

AEOuirfJ) 

S0FT5T 

sqftin 

SQFTAM 

RFACIL(I)  • PSPfl  )*■  f.OFT«-T*;NTSPO*'rrTIN 

• w*tspo*sqft  af* 

Calculate  annual  facility  neq  Bre- 
men ts  'roe*  $0 . Ft  ratlOS  #Or 
students.  Instructor*  and  adminis- 
trators . 

rfacilv : 

IHDFV 

ruinu 

tiegtm 

UIMCTN  • TtFNGH*  I*DDEV*PuIMC  •TlfGTM 

Calculate  development  t'me  'cr 
•nstr  materials  *rqm  de».  rafo  ano 
pet.  of  course  ’enotn  de»  'qd 

i :Nr- 

UPDATE 

INDNNT'l)  . 1'OA’t 

Cal  u5ate  annual  maintenance  time  #or 
i n$ t r materials  from  malnt  update 
ratio. 

;honn‘ ' i : 

supspo 

SUPS TO 

RSL'PPl  ( l ) °SP  i <*  ,.:PSPn*STUDM)#SUPSTO 

Calculate  annua’  supply  rqmts  ‘fir 
student  positions  anc  students 

RSUPPi  . 

NSC5PO 

nscsto 

Rmscd ) • psp' i )«"srsp"*sTUD(i)**scsTp 

Calculate  annual  miscellaneous 
for  student  positions  and  student 

«*.SC( . 
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o Cost  Processing  Logic 


INPUT 

lOGIC 

description 

OUTPUT 

STUDSL 

CSTUD(l)  * ((STUOHk(I)*SMWRRC'I))/52)*STUOSL 

'alculate  annual  student  salary  costs 
in  constant  dol lars . 

CSTUDt I } 

STCST1 

STCST2 

CTRAV(I)  • STU0(1)*(STCSTHSTCST2*(1.0-.5*ARATE) 

Calculate  annual  student  travel  costs 
in  constant  dollars. 

CTRAV(I) 

1HSTSL 

CINSTR(I)  • RINSTR(I)*INSTSL 

Calculate  annual  Instructor  costs  in 
constant  dollars. 

CINSTdd 

AOMSL 

CADHlN(l)  * RAnMlN(l)*AOHSl 

Calculate  annual  administrative 
personnel  costs  in  constant  dollars. 

CAWINdi 

EQUNTC 

CEQOIP(I)  « PEQU1P(I)*E0UNTC 

Calculate  annual  equipment  purchase 
costs. 

CEOUIPdi 

OEPf 

PDfP(L)  * f[DEPF](lOFEQ) 

Calculate  multiplier  of  original  equip- 
ment cost  indiciting  pet . of  deprec'a 
tlon  In  year  i of  life 

PDEP(I) 

POEP(I) 

RVEO  • HCEQUlP(I)*(PDEP(dlOFEQ-N-l)) 
1*1 

for  ( IHOFEQ-N)  >? 

Calculate  the  T-ema’nlnq  value  of  equip- 
ment at  end  of  plannlnq  period. 

RVE* 

cow'd) 

TAEQCU)  • CEOUIP(I)*AEOUIP(J)*COP»»T{I) 

Calculate  the  tota1  annual  equipment 
purchase  and  maintenance  cost 

TAEOCd) 

CPSQfT 

CFACIL(I)  - «FACrL(I)*CPSOn- 

Calculate  annual  facilities  cost  ’r 
constant  dollars 

CF  AC I L ( I 

CIKD 

CIMDKT(I)  - I»»DMNT(I)*C1H0 

Calculate  annual  cost  of  maintaining 
developed  course  materials 

ciHorn ; 

CUIHD  • u:mdtm*cimt 

Calculate  initial  investment  in  In- 
structional materials  development 

Cl  I»c 

EVIH 

RVI*  - CU!M0*EVIM 

Calculate  the  remaining  value  of  in- 
structional materials  at  end  o * 
planning  period 

P,  |H 

CSUPP 

CSUPPL(I)  - RSUPPL(I)*CSUPP 

Calculate  annual  cost  of  supplies 
m constant  dollars. 

CSUPPllI) 

CAIS 

CMISCd)  RMISC ( I )*CM1S 

Calculate  annual  cost  of  miscel- 
laneous In  constant  dollars 

cmscd) 

fACST 

LOff  A 

RFfA  * (LOFfA-tt)*{F*CSJllr>rFtn 

alculate  the  regaining  value  of 

pry* 

facilities  at  the  end  of  tA*  planning 
period 


o Summarization  and  Financial  Factors 


mWJT  LC*0 1 C DESCRIPTION  Ol/tRUT 

DISCNTfl)  ■ f ? . ODRATf) / , 0*(  1 . 0*DftATE )*•! ) Cjlculatf  anoujl  discount  factors.  C 1SCHT(  I , 

INFCAT(t)  ■ (2.0*( 1 , 0» inrate )**1 )/(?.0* INRATE ) Calculate  annual  Inflation.  INFC»T(ll 

N 

NSTUO  ■ T.  CSTUD(I) 

also  NTRMt'v  TbAV': ) ) .MV’R'f  (CINSTP  !1) 

NSUPPl  *f (CADWIN( ! ) ) , 

NS’JPPl  «f  f CSUPPL  ( I ) ) , NMI  SC*f  ( CHI  SC  ( I ) ) 

N 

NFACIl  ■ (Z.CFACIL(I)>FAC0ST 
M 

N 

NEQUIP  • (X.TA*0l  I > - P , ; 

1*1 
N 

ni*o  • (Z  c[HpwT(t))*cur«o-pvrM 

1*1 

N 

OSTUD  • Z CSf U0( I )*01 -CNT  I ' 

1*1 

also  ?tra.  9(ctrav  ;i),:iNSTP*Q'ci»is:pfi)) 

3ADHIN*q(CAWlN< i 

OSUPPl*q(CSliPPL(I  .WJSC'q  “ISC ( l ) ) 

N 

Of  AC  II  • (Z  CFACIini*DISCNTm).fACOt’ 

M 

N 

OEQuip  * Z TAtoC(!)*oisrNi, ; (bvfo.  i o^dratf 

1*1 

**N) 

N 

ofwo  • (Z  cf«wrd!-.ir«csr.  1 1’.  i.i«»'  pvi«  ’O 
1*1 

♦ORATE)***) 
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alculate  non-discounted  total  of  NSTUD 

annual  student  salary,  travel.  In-  *tp*v 

structor  salary,  a<»n1n.  salary,  *r'N$TP 

supply  and  miscellaneous  costs  SACHIN 

Calculate  non-discounted  total  of  NFACIL 

annual  facilities  costs  NSUPPl 

wise 

Calculate  non-discounted  tota'  of  NEOUIP 

annua'  equipment  and  instr.  mat 

costs 

*IW 


Calculate  discounted  total  of  annua'  DSUC 

Student  salary,  travel,  instructor  OTRAV 

salary,  admin  salary,  supply  and  CI*‘ 

miscellaneous  costs  OADAiN 

Calculate  discounted  total  of  annual  DfACIl 

far  ’ ’ ities  costs  p$U>Pl 

wise 

Calculate  discounted  total  o ♦ DEQUIP 

annua'  equipment  and  instructor 
material  costs 


0!«C 


ORATE 

INRATE 
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o 


INPUT 


LOGIC 


DESCRIPTION 


N 

1ST uO  - T.  CSTUD(I)»INFIAT(I) 
1*1 


also  ITRAV-H(CTPAV(I),IINSTP-h(nNSTR(i;  , Calcu’ata  inflated  (current 

I ADMIN*  h (CADMI  N; ; , dollars)  total  o'  annua  student  I'Wrf 

!SUPPl*n(CSUPPL(! ) . IMI  SC=r>  ( CMI  sc  ( I ) salarv.  travel.  Instructor  salary.  UNS’R 

a^in.  salary,  supply  and  mlscel-  IAWIN 

laneous  costs. 

I FAC  1 1 ■ (£_CFACll(I)*INFLAT(I))*FACOST  Calculated  inflated  current  IPAC1L 

l«l  dollars)  total  of  annual  faclll*  ISJPPL 

ties  costs  ^15C 

N 

IEQUIP  « (51  TAEQC(I)*INFLAT(l))-(RVEQ*(l .0*INRATE)**N)  Calculate  m'iated  tota'  of  annua’  IE?  IP 

I«1  equipment  and  instructor  mat.  costs.  11*C 


N 1 

IIHD  - (Z  CINOMT  ( I )*I  NH.  AT  ( I ) )- ( RVIH*( 1 . O*  INRATE  )**N} 
1*1 


N 

FSTUD  * (XL  CSTUD( I )* 1 NFL AT ( ' )*DISCNT ( I ) 

1*1 

also  ftrav»1(CTWV(I)).FINSTR*1(CINSTR(1)),  Calculate  discounted  total  c' 

FADHIN-i (CACHIN' I )).  flated  annual  student  salary,  travel. 

FSJPPL  t1  (CSuPPl  ( I ) ) ,FHISC*1  ( CHI  SC  ( I ) ) instructor  salary  admin  sj  an,  rI*6TR 

supply  and  miscellaneous  costs  rACHIN 

H 

FFACIL  - (C  CFACIL(I)*INaAT:l)“DIS:NT{l)l*FACOST  Caicu'ate  discounted  total  of  in-  ffAClL 

i.l  flated  annual  fac’l’Hes  costs  f$uppl 

PHISC 

FEQUIP  ■ (E.  TAE0C(I)*INFlA’|;)#D1SCNT(I))-(RVE0  Calculated  discounted  total  o'  FEQIIP 

j«1  annual  equipment  and  instructor 

•(1  .CMNRA’f -DRATE  )•**»  "»■*.  COStS  ri"C 


N 

F|MD  = (£  CIMCHlil  i ,•-'!■  s’  : ' -(RVI- 

1*1 

•(1 .0*INRA’f  DRATf  •••. 


SNAOC 

N 

Z NSTUCHN’Bt.»N:s  • fc.st-x  i,.Nf AC.L* 
!■ 

• NHISC’Nt  : P *M**r 

►NSUPPl 

Calculate  total  annual  operating 
costs  non-df scounted,  discounted. 
const  and  cunt  J. 

SNAOC 

SDAOC 

SIAOC 

SFAOC 

also  for  S3AOC •* 1 ' lAOC-f  l 

SFACl-f  » 

r.NAOC 

h 

• SNAOC  '52  GRACl I 

also  GDAOC.  GJAOC  1 

Calculate  average  operating  cost 
per  qraduate,  non-di scounted. 
discounted,  const  and  cumt  S 

GNAOC 

GOACC 

G1AOC 

GFAOC 

PNAOC 

• SNAOC/XPSPC) 

1-1 

also  PDAOC.  PJAOC  i PfAOC 

Calculate  average  operating  cost 
per  student  position. 

Ncn-discounted.  discounted, 
const  and  cumt  I 

PNATC 

PDACC 

PJAOC 

PFAOC 

Non-Discounted  Investment  and 

Cost 

Savings  Vector 

I»UT 

LOGIC 

DESCRIPTION 

OUTPUT 

cnjuip(i) 

FACST 

CUIMD 

ETAIWV(l)  • CEQUIP(l)  ♦ FACST  4 CUIND 

Calculate  first  year  of  Investment 

ETAINVl 1) 

vector 

cfquiP(i) 

ETAIWV(I)  * CEOUIP(I)  for  I * 2.  N - 1 

Calculate  subsequent  years  of 
went  vector  except  for  year  N 

Invest 

FTAIW(I) 

C<QU!P(N) 

■fEQ 

RTVA 

t»IH 

ETAIMV(N)  * CCQUIP(N)  - RVEQ  - RFVA  - 

RVIM 

Calculate  Investment  vector  fp 

r year  N 

1 TAI»»(N) 

AfQWIP(l) 

CORWT(J) 

RFACIl(I) 

CPsgrT 

CIP© 

csn»d) 

cnwr(i) 

CIBSTljJ 

cmppui 

onsc(i) 


f TARf  C(  1 ) = AfQulP(  I )*COPWT(  I ) « RFAClfCPSOn 
! MOUNT  ( I )*f  [NO  4 CSTUD(l)  ♦ CTRA»( 
CINSTR(I)  4 CAOMIN(I)  4 CSUPPl ( I ) 
CMISC(I) 


Calculate  Recurrinq  Cost/$«v1n*t  vecto*  (TAftfC(l) 
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4.  Prepare  output  reports  for  baseline  calculations. 

5.  Retain  non-discounted  Investment  ETAINV*,  and  non-discounted 
Cost/Savings  ETAREC*  vectors  for  completion  of  next  sequence 
of  calculations  with  benefit  pattern  applied. 

6.  Set  up  model  to  sequence  through  calculations  using  input  from 
CMIB  file. 

7.  Read  input  variables  from  first  Entity  Type  1 (Courses)  record. 

8.  Perform  calculations  outlined  in  step  3 for  new  input  variables. 

9.  Print  output  reports  for  calculations  with  benefits  applied. 

10.  Subtract  new  ETAINV*  and  ETAREC*  vectors  from  those  retained  in 

Step  5 to  determine  resultant  Investment  and  Cost/Savings  vectors 
to  be  written  to  the  CMR  file. 

11.  Repeat  sequence  1 through  10  for  remaining  Type  1 Entities  in 
the  CMIA  and  CMIB  files. 

Output:  Outputs  from  the  Training  Cost  Model  are  presented  in  Figures  1 1 1 -28 

through  1 1 1-41 . The  example  outputs  are  for  an  actual  example  which 
was  run  through  the  Cost  Model.  The  upper  part  of  each  page  designated 
course  number  ETA1  represents  a path  through  the  model  without  the 
benefit  applied;  e.g.,  attrition  rate  = .06.  The  lower  part  of  each 
page  designated  as  course  number  ETA2  represents  a path  through  the 
model  with  the  benefit  applied;  e.g.,  attrition  rate  = .04.  The 
following  is  a list  of  the  Training  Cost  Model  outputs: 

1.  Input  Technical  Factors  - Figure  1 1 1-28 

2.  Input  Cost  Factors  - Figure  1 1 1 -29 

3.  Output  Technical  Results  - Figure  1 1 1 -30 

4.  Output  Technical  Results,  Years  1-5  - Figure  1 1 1 -31 
Output  Technical  Results,  Years  6-10  - Figure  1 1 1 -32 
Output  Technical  Results,  Years  11-15  - Figure  1 1 1-33 
Output  Technical  Results,  Years  16-20  - Figure  1 1 1 -34 

5.  Outputs  Cost  Results  - Figure  1 1 1-35 

6.  Output  Cost  Results  (Non-Discounted  Constant  Dollars), 

Years  1-5  - Figure  1 1 1 -36 

Output  Cost  Results  (Non-Discounted  Constant  Dollars), 

Years  6-10  - Figure  1 1 1 - 37 

Output  Cost  Results  (Non-Discounted  Constant  Dollars), 

Years  11-15  - Figure  1 1 1-38 

Output  Cost  Results  (Non-Discounted  Constant  Dollars), 

Years  16-20  - Figure  1 1 1 - 39 

7.  Figure  1 11-40  displays  the  ETAM  Technical  and  Cost  Factors  for 
Years  1-10.  The  final  two  lines  of  this  report  represent  the 
Investment  and  Recurring  Cost/Savings  vectors  which  are  written 
to  the  CMR  file. 

Figure  1 11-41  contains  similar  data  for  years  11-20.  The  data 
on  these  two  figures  represent  the  difference  between  pass  1 
with  the  benefit  not  applied  and  pass  2 with  the  benefit  applied. 

The  previous  reports  will  serve  as  models  for  reports  to  be  generated 
for  Programs  P53  and  P54  which  are  the  vehicles  and  job/tasks  cost 
models. 
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FIGURE  111-31.  TRAINING  COST  MODEL-OUTPUT  TECHNICAL  RESULTS  YEARS  1 - 
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FIGURE  111-37.  TRAINING  COST  MODEL-OUTPUT  COST  RESULTS  YEARS  6 - 10 
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FIGURE  111-38.  TRAINING  COST  MODEL-OUTPUT  COST  RESULTS  YEARS  11 
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FIGURE  111-40.  TRAINING  COST  MODEU-ETAM  OUTPUT  FACTORS  YEARS  1 - 10 
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FIGURE  111-41.  TRAINING  COST  MODEL— ETAM  OUTPUT  FACTORS  YEARS  11-20 
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Program  P53  (Vehicles  Cost  Model). 

Purpose:  The  Vehicles  Cost  Model  calculates  the  delta  cost  effect  from  the 

innovation  benefit  pattern  associated  with  some  instructional  vehicle. 
Since  the  intended  pattern  of  benefits  with  respect  to  instructional 
vehicles  is  one  of  cost  displacement,  the  model  is  relatively  simple. 

The  objective  as  was  with  the  Training  Cost  Model  is  to  create  two 
20  year  dollar  vectors,  one  for  investment  and  a second  for  cost/savings. 

Input:  The  data  input  variables  required  to  run  the  Vehicles  Cost  Model  were 

set  up  by  Program  P51 . The  baseline  input  variables  are  contained  in 
the  Cost  Models  Inputs  W/0  Benefits  (CMIA)  file  and  would  generally 
reflect  the  EXV  file  unless  modified  by  Program  P51 . The  input 
variables  with  benefits  applied  as  set  up  by  Program  P51  are  contained 
in  the  Cost  Models  Input  with  Benefits  (CMIB)  file. 

The  input  variables  are  the  same  as  described  in  Figure  1 1 1 -7  EXV 
DATA  ELEMENTS. 


Process:  The  major  program  process  steps  are: 


1.  Setup  model  to  sequence  through  calculations  using  input  from 
the  CMIA  file. 

2.  Read  input  variables  from  first  Entity  Type  2 (Vehicles)  record. 

3.  Calculate: 


o 


o 


o 


o 


Available  Equipment  annually  over  Planning  Period 

AEQUIP(J)  = £ ± PEQUIP(I) 

J=N  1= J- LOFEQ 

Purchase  Cost  Annually 

CEQUIP(I)  = PEQUIP(I)*  EQUNTC 

Maintenance  Costs  Annually 

MEQUIP(I)  = AEQUIP( I )*C0PMT( I ) 

Installation  Costs  Annually 

IEQUC(I)  = PEQU I P ( I )*EQINSC 


o 


Remaining  Value  of  Equipment 


CEQU I P ( I )*PDEP( LI FERV ) 


for  LIFERV  0 

where  LIFERV  = I + LOFEQ  - N - 1 
and  PDEP  (LIFERV)  = f [DEP0  (LOFEQ) 


Percent  Depreciation  per  Depreciation  Policy 

The  Depreciation  Policies  for  which  PDEP(I)  is  calculated 

are: 
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1)  Linearly  over  life  of  equipment 
PDEP(I)  = I/LOFEQ  for  I = 1 to  LOFEQ 

2)  Sum-of-Digits  over  life  of  equipment 

LOFEQ 

SUM  SUM+1  where  SUM  = 0 initially 

1 = 1 

PDEP(I)  = LOFEQ/SUM  for  I = 1 

PDEP(I)  = ( (LOFEQ- I )/SUM)  + PDEP  (1-1)  for  I = 2 to  LOFEQ 

3)  Fully  depreciated  all  years. 

PDEP(I)  = 1.0  for  I = 1 to  LOFEQ 

o Non-Discounted  Annual  Total  Investment 

IEQUIP(I)  = CEQUIP(I)  + IEQUIP(I) 
for  1=1  , N 

IEQUIP(I)  = TAEQC(I)  - RVEQ 
for  I=N 


o Non  Discounted  Annual  Recurring  Cost/Savings 

REQUIP(I)  = MEQUIP(I)  for  1=1 , N 

o Total  Non-Discounted  Annual  Cost/Savings 

TAEQC(I)  = IEQUIP(I)  + REQUIP(I) 

o Discounted  Annual  Total  Costs  (Constant  Dollars) 

DISCNT(I)  = (2.0  + DRATE )/(2.0*(l .0  + DRATE)**I ) 
for  1 = 1 , N 

DTAEQC(I)  = T AEQC ( I )*DISCNT ( I ) 

o Inflated  Annual  Total  Costs  (Current  Dollars) 

I NF  CAT ( I ) = ( 2 . 0*( 1 . 0+ INRATE** I ) )/ ( 2 . 0+ INRATE) 

ITAEQC(I)  = TAEQC( I )*INFCAT( I ) 

o Discounted  Inflated  Annual  Total  Costs 

FTAEQC ( I ) = I T AEQC ( I )*DISCNT ( I ) 

o Totals  of  Annual  Costs 

N 

TEQUC  = 2Z  TAEQC(I) 

I 

JL. 

DTEQC  = DTAEQC(I) 

1 = 1 


1 11-93 


TAEG  REPORT  NO.  40 


N 

ITEQC  = YZ  ITAEQC(I) 
1=1 


FTEQC  =5^7  FTAECQC(I) 
1 = 1 


4.  Prepare  output  reports  for  baseline  calculations. 

5.  Retain  Non-Discounted  Annual  Total  Investment  I EQU IP*,  and 
Non-Discounted  Annual  Recurring  Cost/Savings,  REQUIP*  vectors 
for  completion  of  next  sequence  of  calculations  with  benefit 
pattern  applied. 

6.  Setup  model  to  sequence  through  calculations  using  input  from 
CM IB  file. 

7.  Read  input  variables  from  first  Entity  Type  2 (Vehicles)  record. 

8.  Perform  calculations  outlined  in  Step  3 for  new  input  variables. 

9.  Print  output  reports  for  calculations  with  benefits  applied. 

10.  Subtract  new  IEQUIP*  and  REQUIP*  vectors  from  those  retained  in 

Step  5 to  determine  resultant  Investment  and  Cost/Savings  vectors 
to  be  written  to  the  CMR  file. 

11.  Repeat  sequence  1 through  10  for  remaining  Type  2 Entities  in 
the  CMIA  and  CMIB  files. 

Outputs:  Outputs  from  the  Vehicle  Cost  Model  program  are  similar  to  the  outputs 

of  Program  P52,  Training  Cost  Model  and  should  include: 

1.  Input  Technical  and  Cost  Factors  combined  report. 

2.  Output  Technical  and  Cost  Results  Combined  report. 

3.  A resultant  Investment  vector  and  a resultant  Recurring  Cost/Savings 
vector  are  written  to  the  CMR  file  for  each  entity  processed. 

Program  P54  (Job/Task  Cost  Model). 

Purpose:  The  Job/Task  Cost  Model  calculates  the  delta  cost  effect  from  the 

innovation  benefit  pattern  associated  with  a change  in  job  performance. 

It  calculates  and  combines  three  major  categories  of  costs;  namely, 
cost  of  equivalent  personnel  for  both  normal  performance  of  the  tasks 
as  well  as  the  performance  required  to  recover  from  certain  task  error 
situations,  the  cost  associated  with  the  commission  of  errors  which  are 
non-manpower  related,  and  the  costs  of  support  items  such  as  tools, 
etc.,  which  are  required  on  an  annual  basis  for  the  performance  of  the 
tasks.  The  objective  as  was  with  the  previous  training  cost  and  vehicles 
cost  models  is  to  create  a twenty-year  dollar  vector.  However,  since 
all  cost/savings  in  the  Job/Task  Model  are  considered  recurring,  no 
Investment  vector  is  created.  A null  investment  vector,  however,  will 
be  written  to  the  CMR  file. 
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Input:  The  data  input  variables  required  to  run  the  Job/Task  Cost  Model  were 

set  up  by  Program  P51 . The  baseline  input  variables  are  contained  in 
the  Cost  Model's  Input  W/0  Benefits  (CMIA)  file  and  would  generally  re- 
flect the  EXJ  file  unless  modified  by  Program  P51 . The  input  variables 
with  benefits  applied,  as  set  up  by  Program  P51 , are  contained  in  the 
Cost  Model's  Input  With  Benefits  (CMIB)  file. 

The  input  variables  are  the  same  as  described  in  Figure  1 1 1 -8  EXJ 
DATA  ELEMENTS. 

Process:  The  major  program  process  steps  are: 

1.  Set  up  model  to  sequence  through  calculations  using  input 

from  the  CMIA  file. 

2.  Read  input  variables  from  first  Entity  Type  3 (Job/Tasks)  record. 

3.  Calculate: 

o Annual  Task  Time  Without  Errors  (by  Type  Person) 

OFFTT(I)  = APERS( I )*TTASK*FTASK*OPTSK 
for  I = 1 to  N 

ENLTT(I)  = APERS ( I ) *ITASK*FTASK*EPTSK 
for  I = 1 to  N 

CIVTT(I)  = APERS( I )*TTASK*FTASK*CPTSK 
for  1=1  to  N 

TOTTT(I)  = OFFTT(I)  + ENLTT(I)  + CIVTT(I) 
for  1=1  to  N 

o Annual  Error  Time 

ERRTT(I)  = APERS ( I )*TTASK*FTASK*ERTSK 
for  I = 1 to  N 

o Total  Annual  Task  Time  (by  Type  Person) 

TTOFF(I)  = OFFTT(I)  + (ERRTT ( I )*EPTSK) 
for  I = 1 to  N 

TTENL(I)  = ENLTT(I)  (ERRTT(  I )*EPTSK) 
for  I = 1 to  N 

TTCIV(I)  = CIVTT(I)  + (ERRTT( I )*CPTSK) 
for  1=1  to  N 

TTTOT(I)  = TTOFF(I)  + TTENL(I)  + TTCIV(I) 
for  I = 1 to  N 

o Total  Annual  Equivalent  Persons  (by  Type  Person) 

EQVOFF ( I ) = TTOFF ( I )/2080 
for  1=1  to  N 

EQVENL ( I ) = TTENL ( I )/2080 
for  1=1  to  N 

EQVCI V ( I ) = TT  C I V ( I ) / 2080 
for  I = 1 to  N 

EQVTOT ( I ) = EQVOFF ( I ) + EQVENL ( I ) + EQVCI V( I ) 
for  I = 1 to  N 
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o Annual  Salary  Cost  (by  Type) 

OFFSLC(I)  = EQVOFF( I )*0FFSL 
for  I = 1 to  N 

ENLSLC(I)  = EQVENL( I )*ENLSL 
for  I = 1 to  N 

CIVSLC(I)  = EQVC I V ( I )*CIVSL 
for  I = 1 to  N 

TOTSLC(I)  = OFFSLC(I)  + ENLSLC(I)  + CIVSLC(I) 
for  I = 1 to  N 

o Total  Annual  Costs  (by  Type) 

TOFFC(I)  = OFFSLC(I)  + (0FFTT( I )/TTASK)*OERRC  + APERS(I)* 
SUPPC*OPTSK 
for  I = 1 to  N 

TENLC(I)  = ENLSLC(I)  + ( ENLTT ( I )/TTASK)*OERRC  + APERS(I)* 
SUPPC*EPTSK 
for  I = 1 to  N 

TCIVC(I)  = CIVSLC(I)  + (Cl VTT ( I )/TTASK)*OERRC  + APERS(I)* 
SUPPC*CPTSK 
for  1=1  to  N 

o Non-Discounted  Annual  Recurring  Cost/Savings 

TTOTC(I)  = TOFFC(I)  + TENLC(I)  + TCIVC(I) 
for  1=1  to  N 

o Discounted  Annual  Recurring  Cost/Savings  (Constant  Dollars) 

DISCNT(I)  = (2.0  + DRATE)/(2.0*(1.0+DRATE)**I) 
for  I = 1 , N 

DTTOTC(I)  = TT0TC( I ) *D I SCNT ( I ) 
for  I = 1 , N 

o Inflated  Annual  Recurring  Cost/Savings  (Current  Dollars) 

INFCAT(I)  = (2.0*(1 .0  + INRATE** I ) )/ ( . 20  + INRATE) 
for  I = 1 , N 

ITTOTC(I)  = TT0TC( I ) * I NFCAT ( I ) 
for  I = 1,  N 

o Discounted  Inflated  Annual  Costs 

FTTOTC(I)  = ITT0TC( I )*DISCNT( I ) 
for  I = 1 , N 

o Totals  of  Annual  Costs 

N 

TTOC  =7"  TTOTC(I) 
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N 

DTOC  =T~  DTTOTC(I) 
1 = 1 

HOC  = > ITTOTC(I) 
1 = 1 
N 

FTOC  =ZT  FTTOTC(I) 
1 = 1 


4.  Prepare  output  reports  for  baseline  calculations. 

5.  Retain  non-di scounted  annual  recurring  cost/savings,  TTOTC* 
vector  for  completion  of  next  sequence  of  calculations  with 
benefit  pattern  applied. 

6.  Set  up  model  to  sequence  through  calculations  using  input  from 
CMIB  file. 

7.  Read  input  variables  from  first  Entity  Type  3 (Job/Tasks)  record. 

8.  Perform  calculations  outlined  in  Step  3 for  new  input  variables. 

9.  Print  output  reports  for  calculations  with  benefits  applied. 

10.  Subtract  new  TTOTC*  vector  from  the  previous  one  retained  in 
Step  5 to  determine  resultant  Cost/Savings  vector  to  be  written 
to  the  CMR  file.  Since  no  Investment  vector  was  calculated,  a 
null  vector  will  be  written  to  the  CMR  file  for  investment. 

11.  Repeat  sequence  1 through  10  for  remaining  Type  3 Entities  in 
the  CMIA  and  CMIB  files. 


Outputs:  Outputs  from  the  Job/Task  Cost  Model  program  are  similar  to  the  outputs 

of  Program  P52,  Training  Cost  Model,  and  should  include: 

1.  Input  Technical  and  Cost  Factors  Combined  Report. 

2.  Output  Technical  and  Cost  Results  Combined  Report. 

3.  A resultant  Recurring  Cost/Savings  vector  and  a null 
Investment  vector  are  written  to  the  CMR  file  for  each 
entity  processed. 

Cost  Model  Summary.  Each  of  the  cost  models  writes  a record  to  the  CMR  file  per 
the  format  previously  identified  for  that  file.  The  final  step  of  the  Cost 
Models  program  should  summarize  the  results  for  all  entities.  CMR  file  Record 
Type  5 (Pos  1 = 5)  will  contain  the  summary  of  all  Courses,  Record  Type  6 will 
contain  a summary  of  all  Vehicles,  and  Record  Type  7 will  contain  a record  of 
all  Job/Tasks.  The  total  of  these  three  summaries  will  be  added  into  a Record 
Type  8.  The  Record  Type  9 of  the  CMR  file  will  be  summarized  into  by  Program 
PGO  after  the  Risk  Package  cost  for  Record  Type  4 has  been  identified. 

ETAM  RISK  REDUCTION.  A major  aspect  of  the  ETAM  process  is  the  evaluation  of 
various  alternatives  to  implementing  and  gaining  user  acceptance  of  the  innovation. 
A number  of  risk  factors  with  regard  to  the  acceptance  of  the  innovation  were  pre- 
viously identified  and  their  profile  was  stored  in  the  RKP  file.  An  example  of 
this  file  was  previously  shown  in  Figure  1 1 1 - 1 0 . As  a result  of  the  risks  that 
were  identified,  a number  of  risk  reduction  projects  were  devised.  The  projects 
were  targeted  at  the  high  risk  categories  outlined  in  the  RKP  file.  The  RRPJ 
file  contains  descriptions  of  the  risk  projects.  Figure  1 1 1 - 11  showed  an  example 
of  the  RRPJ  file  contents.  The  assessor  can  group  these  projects  into  up  to  nine 
combinations  for  subsequent  evaluation  within  the  Decision  Analysis  part  of  the 
ETAM  Assessment  function.  The  objectives  of  the  Risk  Reduction  function  are: 
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1.  Permit  the  assessor  to  review  risk  data  stored  in  the  RKP  and  RRPJ 
files. 

2.  Provide  the  user  with  the  capability  of  selecting  up  to  nine  combi- 
nations  of  the  projects  stored  in  the  RRPJ  file  and  to  retain  these 

groupings'  trr-ttre-RftP-K file  for  subsequent  access  by  the  Decision 
Analysis  function. 

3.  Provide  the  user  with  access  to  the  program  for  assigning  probabilities 
of  implementation  success  and  user  acceptance  success  to  the  various  risk 
packages.  This  is  performed  by  calling  Program  P59,  which  will  also  be 
accessible  to  the  user  under  the  Decision  Analysis  function. 


These  objectives  are  accomplished  through  two  programs,  namely: 

P56  - Risk  Reduction  Project  Grouping 
P5S  - Assign  Probabilities 

These  programs  access  three  Project  Data  Base  files: 


RKP  which  stores  a profile  of  the  risks  associated  with  acceptance  of 
the  innovation, 

RRPJ  which  stores  any  of  the  identified  Risk  Reduction  projects,  and 
RRPK  which  will  contain  the  final  results  of  the  project  groupings  as 
performed  by  Program  P56. 

The  formats  for  the  data  files  were  previously  outlined  in  the  section  on  the 

Project  Data  Base.  In  this  section,  the  program  to  support  the  grouping  of  the 

Risk  Reduction  projects  is  outlined,  namely  Program  P56.  Figure  1 1 1 -42  shows 

the  interrelationships  between  the  Programs  P56,  P59,  and  the  Executive  Control. 

Program  P56  (Risk  Reduction  Project  Grouping). 

Purpose:  This  program  permits  the  user  to  review  the  various  descriptive  and 

risk  project  data  and  then  to  group  the  identified  projects  into 
combinations  for  storage  in  the  RRPK  file.  The  user  has  the  option 
of  inputting  probabilities  directly  to  the  RRPK  file  as  the  Risk 
Reduction  projects  are  grouped,  or  of  calling  the  Assign  Proba- 
bilities Program  P59  in  order  to  assist  in  establishing  the  proba- 
bilities. The  user  can  continue  to  group  Risk  Reduction  projects 
until  either  the  QUIT  option  is  selected  or  a maximum  of  nine  project 
groupings  have  been  made  for  storage  in  the  RRPK  file. 

Inputs:  The  major  inputs  to  Program  P56  are  the  descriptive  data  in  the  RKP 

and  RRPJ  files  as  well  as  the  cost  data  stored  in  the  RRPJ  file  associ- 
ated with  each  Risk  Reduction  project.  The  format  of  these  files  has 
been  previously  described  in  the  Project  Data  Base  part  of  this  section 
of  the  report. 

Process:  The  major  program  process  steps  are: 

1.  Provide  access  to  Program  P56  through  either  the  RUNRISK  EXEC 

or  the  ETAM  EXEC,  both  of  which  call  the  RISK  EXEC  which  accesses 
Program  P56. 
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2.  Select  options  are  displayed  which  permit  the  user  to 

o Group  Risk  Reduction  projects 

o Assign  probabilities 

o Review  RKP  file 

o Review  RRPJ  file 

o Quit 

3.  The  user  is  permitted  to  select  one  of  these  options  from  a 
terminal  and  is  branched  to  the  appropriate  processing  function 
for  that  option.  In  the  case  of  requests  for  reviewing  the 
RKP  and  RRPJ  files.  Program  PI  is  called.  Following  completion 
of  the  descriptive  listings,  the  user  will  be  returned  to  the 
select  options  display  for  another  choice. 

4.  If  the  user  selects  the  Assign  Probabilities  option.  Program 
P59  will  be  called,  after  which  the  user  will  be  returned  to 
the  select  display  option.  If  the  user  selects  the  Group  Risk 
Reduction  Projects  option,  the  Program  P56  processing  steps  will 
continue. 

5.  Program  P56  will  access  the  RRPJ  file  and  will  list  the  descriptive 
zero  records.  These  records  contain  the  project  ID,  project  name, 
date,  and  overall  project  cost. 

6.  The  user  is  then  requested  to  select  one  or  more  of  the  Risk  Reduc- 
tion projects  within  the  first  grouping.  This  is  performed  by  enter- 
ing the  project  ID;  e.g.,  001-ATTD,  001-SART,  etc.  If  the  user 
wishes  to  enter  probabilities  at  this  time,  the  last  project  ID  in 
the  grouping  should  be  followed  with  a slash  (/)  and  then  nine 
two-digit  probabilities  separated  by  commas.  Also,  the  project 

cost  which  appeared  in  the  RRPJ  file  descriptive  record  zero  can 
be  entered  immediately  after  the  ninth  probability.  Figure  III-12 
showed  an  example  of  entries  in  the  RRPK  file.  For  example,  the 
entry  appearing  opposite  record  1-0  would  be  inputted  as  follows: 
001-SART,  002-SART,  003-SART,  004-SART,001 -ATTD/90,95,99,80,85,90, 
50,60,70,110550000.  The  project  ID's  prior  to  the  slash  would  be 
placed  in  record  1-0  of  the  RRPK  file  while  the  probabilities  and 
costs  after  the  slash  would  appear  opposite  the  1-1  record  of  the 
RRPK  file. 

7.  The  number  of  groups  is  checked  and  if  greater  than  nine,  the  user 
will  be  exited  to  the  ETAM  EXEC.  If  less  than  nine,  the  user  will 

be  requested  if  additional  Risk  Reduction  project  groups  are  desired. 
If  the  user  selects  this  option,  another  grouping;  i.e.,  group  2 
can  be  entered  via  the  terminal. 

8.  Final  results  will  be  stored  in  the  RRPK  file. 

Outputs:  The  ouput  of  Program  P56  is  one  or  more  records  to  be  stored  in  the 

RRPK  file  identifying  project  groups,  associated  probabilities,  and 

cost. 

ETAM  DECISION  ANALYSIS.  The  DTREE  routines  sequence  and/or  permits-  assessor 
selection  of  the  programs  required  to  develop  decision  tree  input  factors  and  to 
perform  the  calculations  which  will  allow  generation  of  the  ETAM  DECISION  ANALYSIS 
DETAIL  REPORT  (DT01).  Minimal  requirements  for  performing  the  decision  tree  cal- 
culations are  the  development  of  outcome  utilities  and  node  probabilities.  The 
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Decision  Analysis  report  DT01  will  be  generated  if  1)  probab i 1 i ties  exist  in  the 
number  0 record  (NO  RISK  PK6)  of  the  RRPK  file,  and  2)  outcome  utilities  exist 
in  the  number  999  record  of  the  VARF  file.  If  new  or  additional  probabilities 
are  to  be  developed,  P59  is  called  to  prompt  the  assessor  in  specifying  proba- 
bilities. If  new  or  modified  outcomes  are  to  be  developed,  P58  is  called  to 
permit  the  assessor  to  1)  select  new  relevant  variables  from  the  VQAL  project 
file,  and  2)  to  prompt  the  assessor  in  the  process  sequence  for  assigning  values  to 
the  selected  variables.  Several  display  formats  guide  the  assessor  in  developing 
and  validating  the  decision  tree  outcomes.  Figure  III -43  shows  the  process  flow 
for  the  Decision  Analysis  routines  within  the  ETAM  Assessment  function. 


The  basic  methodology  in  performing  the  ETAM  Decision  Analysis  centers  around 
the  decision  tree  which  is  shown  in  Figure  1 11-44.  Each  alternative  path  in  the 
tree  leads  to  an  outcome  value  (A,  B,  C & D)  which  incorporates  both  quantitative 
Cost  Model  Results  (from  CMR  project  file)  and  equivalent  dollar  (utility) 
values  for  relevant  qualitative  factors.  Probabilities  that  an  outcome  will 
occur  are  estimated  at  PX  & PW).  The  probability  estimates  are  developed  and 
stored  in  the  variable  reference  (VARF  project  file).  Risk  patterns  which 
reduce  the  success  probabilities  can  be  moderated  by  undertaking  Risk  Reduction 
Projects  (RRPJ  project  file)  which  have  been  grouped  into  Risk  Reduction  Packages 
(RRPK  project  file).  The  decision  tree  provides  for  up  to  nine  (9)  Risk  Reduction 
Packages  with  their  associated  costs  (C0ST-C0ST9)  and  modified  success  probabil ities 
(PX1-PX9,  PW1-PW9).  The  probability  of  Risk  Reduction  Project  success  can  also 
be  estimated  (PI1-PI9).  There  are  outcome  probabilities  (PZ1-PZ9  & PV1-PY9) 
associated  with  the  failure  of  the  projects  which  are  also  estimated. 


Considering  this  decision  analysis  framework,  the  major  objectives  in  performing 
the  decision  analysis  are  to: 

1.  Develop  values  for  the  four  major  potential  outcomes  resulting  from 
the  assessment  of  the  innovation;  i.e.,  A)  acceptance,  successful 
implementation,  user  acceptance;  B)  acceptance,  successful  implemen- 
tation, user  non-acceptance;  C)  acceptance,  inability  to  implement; 
and  D)  rejection. 

2.  Estimate  probabilities  for  each  of  the  potential  outcomes. 

3.  Develop  an  accept/reject  decision  variable  based  on  the  outcome 
values  and  the  probabilities  of  their  occurrence. 

4.  Introduce  various  combinations  of  risk  reduction  packages  into 
the  decision  process,  considering  their  costs  and  effects  upon  the 
success  probabi 1 i ties  of  the  outcomes. 

The  first  objective  is  accomplished  by  taking  the  quantitative  results  of  applying 
the  appropriate  cost  model  to  the  benefit  pattern,  and  developing  equivalent 
values  for  each  of  the  identified  qualitative  variables  which  are  stored  in  the 
VQAL  project  file.  A relative  importance  factor  is  identified  for  the  cost/savings 
variable  as  well  as  each  of  the  qualitative  variables.  A value  can  then  be 
developed  for  each  outcome  which  considers  the  independent  value  of  each  variable 
and  its  relative  importance.  The  outcome  scenarios  previously  developed  and 
stored  in  the  SCEN  project  file  are  reviewed  by  the  assessor  during  this  variation 
process . 
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FIGURE  111-44.  DECISION  TREE  FOR  ASSESSMENT  OF  TRAINING  INNOVATION 
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The  second  objective  is  achieved  by  considering  the  various  risks  in  implemen- 
tation and  in  user  acceptance  which  were  previously  identified  descriptively  in 
the  RKP,  Risk  Profile  summary  project  file.  An  initial  set  of  probabilities  was 
developed  at  the  time  the  various  risk  reduction  projects  were  grouped,  since  it 
was  necessary  at  that  time  to  modify  those  probabilities  considering  the  effects 
of  the  projects  upon  implementation  success  and  user  acceptance.  At  this  time 
the  probabilities  should  be  reviewed  and  modified  if  appropriate.  The  third 
objective  is  accomplished  by  calculating  the  expected  value  of  the  accept  path 
using  the  outcome  values  and  probabilities.  Since  a reject  decision  has  a zero 
value,  any  positive  expected  value  on  the  accept  decision  leg  would  generally 
indicate  that  the  innovation  has  benefit  and  should  be  accepted.  The  calculation 
of  a number,  however,  is  no  panacea  for  the  difficult  decision  process  which 
requires  a constant  examination  of  all  assumptions  which  went  into  the  derivation 
of  the  final  number.  Also,  it  should  be  remembered  that  the  decision  rule  for 
calculating  the  decision  variable  was  expected  value.  If  the  decision  maker 
intends  to  apply  another  rule,  such  as  minimization  of  risk,  then  the  result 
will  have  to  be  reexamined  in  that  light. 

The  final  objective  is  to  modify  the  decision  components  with  the  introduction 
of  projects,  which  at  some  cost  can  improve  the  success  probability  of  intro- 
ducing the  innovation  into  the  system.  The  risk  reduction  projects  documented 
in  the  RRPJ  project  file  were  previously  grouped  and  the  groupings  along  with 
their  effects  upon  the  success  probabilities,  were  stored  in  the  RRPK,  Risk 
Reduction  Packages  project  file.  A new  decision  variable,  for  each  path  with  a 
risk  reduction  package,  can  be  calculated.  The  decision  variable  will  have 
teen  calculated  considering  the  costs  of  the  risk  reduction  packages.  Now  the 
path  with  the  highest  positive  expected  value  will  generally  be  the  most  accept- 
able, subject  to  the  previous  caveats. 

Several  display  formats  are  used  in  aiding  the  assessor  in  estimating  magnitudes 
and  importance  weightings  of  the  variables  considered  in  the  outcome  utility. 
Additional  reports  present  the  results  of  the  ETAM  Decision  Analysis  process. 

This  section  begins  with  the  ETAM  DECISION  ANALYSIS  DETAIL  REPORT  (DT01)  which 
describes  the  inputs,  calculations,  and  outputs  related  to  the  decision  tree 
structure  previously  outlined.  Following  the  detail  report  are  the  major  selec- 
tive interpretive,  logical,  and  calculational  sequences  for  the  assessor  to 
assign  values  and  importance  weightings  to  variables,  and  to  associate  probabi- 
lities with  the  various  decision  tree  paths.  These  functions  are  accomplished  by 
Program  P58  and  P59,  respectively.  Program  P60  controls  the  Decision  Analysis 
sequence  and  performs  the  calculations  required  to  produce  the  DT01  Report. 

ETAM  Decision  Analysis  Detail  Report  (DT01).  The  following  identifies  data 
sources  and  calculational  logic  for  generating  the  fields  on  the  ETAM  DECISION 
ANALYSIS  DETAIL  REPORT  which  is  produced  by  Program  P60.  Fields  outlined  rn  the 
report  layout  not  described  here  are  fixed,  and  are  generated  by  the  program 
format  statements.  A sample  report  showing  data  elements  by  column/row  coordinate 
is  shown  in  Figure  1 1 1 -45 . Up  to  three  reports  may  be  required  if  the  total  of 
nine  maximum  risk  reduction  packages  is  used.  The  NO  RISK  PKG  column  of  data 
elements  is  repeated  on  each  report. 
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A 


h. 


COL/ ROW  VARIABLE 
IB  - DATE 


2B  - RPTNO 


3B  - NORPTS 


2D  - RKPKGX 

3D  - RKPKGY 

4D  - RKPKGZ 


IE  - RANKA 

2E  - RAN KB 

3E  - RANKC 

4E  - RANKD 


1 F - OUTCMA 
2F  - OUTCMA 
3F  - OUTCMA 
4F  - OUTCMA 

1G  - OUTCMB 
2G  - OUTCMB 
3G  - OUTCMB 
4G  - OUTCMB 

1H  - PX+'NA’ 


2H  - PXA+PZA 
3H  - PXB+PZB 
4H  - PXC+PZC 


DATA  SOURCE  AND/OR  CALCULATIONAL  LOGIC 


This  field  is  generated  within  the  system  and 
developed  as  MM/DD/YY. 


This  field  contains  the  number  of  the  report  being 
printed  (1 , 2 or  3) . 

This  field  gives  the  total  number  of  DT01  reports 
that  are  generated.  With  0-3  risk  pages,  this  will 
be  1,  4-6  will  be  2,  and  7-9  will  be  3. 


This  field  contains  the  risk  package  number  for  which 
the  evaluation  below  it  is  made.  On  Report  1 these 
values  will  be  1 , 2 and  3;  on  Report  2 they  will  be 
4,  5 and  6,  and  on  Report  3 they  will  be  7,  8 and  9. 

This  field  will  contain  a number  between  1 and  0,  where 
a 1 indicates  the  highest  ranked  alternative;  i.e.,  the 
Value  of  the  Decision  Variable  (in  Row  Z)  which  is  the 
most  positive,  and  a 0 indicates  the  lowest  ranked 
alternative.  Interim  values  2 through  9 will  be  assigned 
accordingly.  In  cases  where  equal  values  of  the  decision 
variable  exist,  available  numbers  will  be  sequenced  from 
left  to  right. 

These  fields  contain  the  summation  of  the  outcomes, 
weighted  variable  equivalent  dollar  values  which  are 
are  stored  in  the  number  999  record  of  the  VARF 
project  file. 

These  files  contain  the  Outcome  B variable  equi- 
valent dollar  total  from  the  VARF  project  file 
(See  1F-4F  description). 


This  field  contains  the  decision  tree  probability  at 
the  X node  for  the  condition  where  the  innovation 
is  accepted  without  need  to  undertake  any  risk  re- 
duction projects.  It  is  developed  interactively  (or 
entered  in  batch)  and  stored  in  the  number  0 record 
of  the  RRPK  project  file.  Since  no  projects  are 
considered  in  this  decision  path,  the  second  proba- 
bility (P2)  is  not  applicable  which  is  indicated  by 
the  NA. 

This  field  contains  the  probabilities  at  the  X1-X9 
and  Z1-Z9  nodes  which  reflect  incorporation  of  risk 
reduction  projects  into  the  decision  process.  PXA 
will  contain  the  value  of  PX1 , PX4,  or  PX7  depending 
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1J 

2J 

3J 

4J 


IK. 


2K 

3K 

4K 


1M 

2M 

3M 

4M 

IN 


2N 

3N 

4N 


upon  the  Report  Number  (2B-RPTN0)  being  generated  (PXB 
will  be  PX2 , PX5  or  PXB,  and  PXC  will  be  PX3,  PX6  or 
PX9).  PZA  will  contain  the  value  of  PZ1 , PZ4,  or  PZ7, 
also  depending  upon  the  report  number  (PZB  will  be  PZ2, 

PZ5,  or  PZ8,  and  PZC  will  be  PZ3,  PZ6  or  PZ9).  These 
values  are  stored  in  number  1 through  9 records  (reflec- 
ting data  on  Risk  Packages  1 through  9 respectively)  of 
the  RRPK  project  file. 

X This  is  the  expected  value  at  the  X,  XI,  X2  . . . X9 

XA  nodes  of  the  decision  tree.  XA  will  reflect  XI,  X4, 

XB  or  X7  depending  upon  the  report  number  (XB  will  be  X2, 

XC  X5  or  X8  and  XC  will  be  X3,  XC,  or  X9).  The  value  is 

calculated  as: 

Xn  = (PXn  * OUTCMA  + (1  - PXn)  * OUTCMB) 
where  n is  blank,  1,2,  . . .,9. 

' NA 1 No  path  exists  for  the  NO  RISK  PKG  situation,  therefore, 

this  field  is  identified  as  not  applicable  (NA). 

ZA  This  is  the  expected  value  at  the  Z1 , Z2,  . . .,79  nodes 

ZB  of  the  decision  tree.  ZA  will  reflect  Z1 , Z4,  or  Z7 

ZC  depending  upon  the  report  number.  (ZB  will  be  Z2,  Z5  or 

Z8,  and  ZC  will  be  Z3,  Z6,  or  Z9).  The  value  is  calculated 
as: 

Zn  = (PZn  * OUTCMA  + (1  - PZn)  * OUTCMB) 
where  nisi, 2,  . . .,9. 

OUTCMC  These  fields  contain  the  Outcome  C variable  equi- 

OUTCMC  valent  dollar  total  from  the  VARE  project  file 

OUTCMC  (See  1F-4F  description). 

OUTCMC 

PW+'NA'  This  field  contains  the  decision  tree  probability 

at  the  W node  for  the  condition  where  the  innovation 
is  accepted  without  need  to  undertake  any  risk  reduction 
projects.  It  is  developed  interactively  (or  entered  in 
batch)  and  stored  in  the  number  0 record  of  the  RRPK 
project  file.  Since  no  projects  are  considered  in  this 
decision  path,  the  second  probability  (PY)  is  not 
applicable  as  indicated  by  the  NA. 

PWA+PYA  This  field  contains  the  probabilities  at  the  W1-W9  and 
PWB+PYB  Y1-Y9  nodes  which  reflect  incorporation  of  risk  reduction 
PWC+PYC  projects  into  the  decision  process.  PWA  will  contain 

the  value  of  PW1 , PW4,  or  PW7  depending  upon  the  report 
number  (PWB  will  be  pW2,  PW5,  or  PW8,  and  PWC  will  be 
PW3,  PW6,  or  PW9).  PYA  will  contain  the  value  of  PY1 , PY4, 
or  PY7,  also  depending  upon  the  report  number  (PYB  will 
be  PY2 , PY 5 , or  PY3,  and  PYC  will  be  PY3 , PY6,  or  PY9) . 
These  values  are  stored  in  number  1 through  9 records  of 
the  RRPK  project  file. 
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IP 

2P 

3P 

4P 


IQ 


2Q 

30 

4Q 


IS 

2S 

3S 

4S 


IT 

2T 

3T 

4T 


1U 


21) 

3U 

4U 


IV 


W This  is  the  expected  value  at  W,  WT , W2,  . . . , W9  nodes 

WA  of  the  decision  tree.  WA  will  reflect  W1 , W4,  or  W7 

WB  depending  upon  the  report  number  (WB  will  be  W2,  W5,  or 

WC  W8,  and  WC  will  be  W3,  W6,  or  W9).  The  value  is  calculated 

as: 

Wn  = (PWn  * Xn  + (1  - PWn)  * OUTCMC) 
where  n is  blank,  1,2,  . . .,9. 

' NA*  No  path  exists  for  the  NORISK  PKG  situation,  therefore, 

this  field  is  identified  as  not  applicable  (NA). 

YA  This  is  the  expected  value  at  the  Y 1 , Y2,  . . . , Y9 

YB  nodes  of  the  decision  tree.  YA  will  reflect  Y 1 , Y4,  or 

YC  Y7  depending  upon  the  report  number  (YB  will  be  Y2,  Y5, 

or  Y8,  and  YC  will  be  Y3,  Y6,  or  Y9).  The  value  is 
calculated  as: 

Yn  = ( PYn  * Zn  + (1  - PYn)  * OUTCMC) 
where  n is  1 , 2,  . . . , 9. 

OUTCMD  These  fields  contain  the  Outcome  D variable  equivalent 

OUTCMD  dollar  total  from  the  VARF  project  file.  (See  1F-4F 

OUTCMD  description).  This  field  will  generally  be  all  zeros 

OUTCMD  since  there  should  neither  be  benefits  nor  liabilities 

associated  with  the  reject  path. 

DECO  These  fields  will  contain  the  decision  variable  prior 

PSUCCA  to  subtracting  out  the  Risk  Reduction  Package  costs.  In 

PSUCCB  the  case  of  the  NO  RISK  PKG  leg,  the  value  of  the  final 

PSUCCC  decision  variable  (DECO)  is  printed  so  it  can  be  compared 

with  the  same  factor  for  the  risk  package  paths.  PSUCCA 
will  contain  PSUCC1 , PSUCC4,  or  PSUCC7,  depending  upon 
the  report  number  (PSUCCB  will  be  PSUCC2,  PSUCC5,  or 
PSUCC8,  and  PSUCCC  will  be  PSUCC3,  PSUCC6,  or  PSUCC9) . 

The  value  is  determined  by  taking  the  more  positive  of  Wn 
or  OUTCMD  (generally  zero). 

'NA'  No  path  exists  for  the  NO  RISK  PKG  situation,  therefore, 

this  field  is  identified  as  not  applicable  (NA). 

PFAILA  These  fields  will  contain  the  decision  variable  prior 

PFAILB  to  subtracting  out  the  Risk  Reduction  Package  costs 

PFAILC  PFAILA  will  contain  PFAIL1 , PFAIL4,  or  PFAIL7  depending 

upon  the  report  number  (PFAILB  will  be  PFAIL2,  PFAIL5,  or 
PFAIL8,  and  PFAILC  will  be  PFAIL3,  PFAIL6,  or  PFAIL9) . 

The  value  is  determined  by  taking  the  more  positive 
of  Yn  or  OUTCMD  (generally  zero). 

'NA'  Probability  of  project  success  is  not  applicable  to  the 

NO  RISK  PKG  path,  therefore  it  is  shown  as  not  applicable 
(NA). 
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2V  - PIA  This  field  contains  the  probabilities  that  the  Risk 

3 V - PIB  Reduction  Package  in  that  decision  path  will  be 

4V  - PIC  successfully  completed.  PIA  will  contain  the  value 

of  PI1,  P 14 , or  PI7  depending  upon  the  report  number 
(PIB  will  contain  PI2,  PI5,  or  PIB,  and  PIC  will  contain 
PI3,  PI6,  or  PI9).  These  values  are  stored  in  number 
1 through  9 records  of  the  RRPK  project  file. 

1W  - ’ NA * The  value  of  the  risk  package  is  not  applicable  in  the 

NO  RISK  PKG  path,  therefore  it  is  snown  as  NA. 

2W  - RKPKGA  This  is  the  expected  value  of  the  decision  tree  paths 

3W  - RKPKGB  to  which  Risk  Reduction  Packages  have  been  applied. 

4W  - RKPKGC  RKPKGA  will  contain  RKPKG1 , RKPKG4,  or  RKPKG7  depending 

upon  the  report  number.  (RKPKGB  will  be  RKPKG2,  RKPKG5, 
or  RKPKG8 , and  RKPKGC  will  be  RKPKG3 , RKPKG6 , or  RKPKG9 ' 
The  value  is  calculated  as: 

RKPKGn  = (Pin  * PSUCCn  + (1  - Pin)  * PFAIln) 
where  n is  1 , 2,  . . . , 9. 

IX  - 'NA1  The  cost  of  the  Risk  Reduction  Package  is  not  appli- 

cable to  the  NO  RISK  PKG  path,  therefore  it  is  shown 
as  NA. 

2X  - COSTA  These  fields  contain  the  value  of  the  Risk  Reduction 

3X  - COSTB  Package  in  that  particular  decision  tree  path.  At 

4X  - COSTC  the  time  the  risk  projects  were  grouped  and  identified 

in  the  RRPK  project  file,  their  individual  costs  were 
added  and  stored  in  that  file.  COSTA  will  contain 
C0ST1 , C0ST4,  or  C0ST7  depending  upon  the  report  number 
(COSTB  will  be  C0ST2,  C0ST5,  or  C0ST8  and  COSTC  will 
be  C0ST3,  COSTC,  or  C0ST9).  The  respective  record 
number  in  the  RRPK  project  file  should  be  accessed  to 
obtain  this  value.  When  the  Decision  Analysis  has 
been  completed  and  the  decision  path  selected,  the 
respective  COSTn  value  in  that  path  will  be  stored  in 
the  CMR  project  file  (Type  4 record)  for  input  to  the 
Financial  Analysis  routine. 

1Z  - DECO  These  fields  contain  the  final  value  on  the  decision 

2Z  - DECA  variable  for  the  respective  decision  path.  DECO  will 

3Z  - DECB  have  the  same  value  previously  determined  in  IT.  DECA 

4Z  - DECC  contain  DEC 1 , DEC4,  or  DEC7  depending  upon  the  report 

number  (DECB  will  be  DEC2,  DECS,  or  DEC8,  and  DECC  will 
be  DEC3,  DEC6,  or  DEC9).  This  value  is  calculated  as: 

DECn  = RKPKGn  - COSTn 

where  n is  1 , 2,  . . . , 9. 
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DTOl  Variable 

List  Sunmary. 

VAR  NAME 

I/O 

SOURCE/DESTINATION 

REPORT  REF 

DATE 

0 

INT  GEN/TRER  + RPT 

IB 

RPTNO 

0 

INT  GEN/RPT 

2B 

NORPTS 

0 

INT  GEN/RPT 

3B 

RKPKGX-Z 

0 

INT  GEN/TRER  + RPT 

2D-4D 

RANKA-D 

0 

CALC/TRER  + RPT 

IE-4E 

OUTCMA-D 

I 

VARF/TRER  + RPT 

1F-4F,  1G- 

1S-4S 

PXb-9 

I 

RRPK/TRER  + RPT 

IH-4H 

PZ1-9 

I 

RRPK/TRER  + RPT 

2H-4H 

XU-9 

0 

CALC/TRER  + RPT 

1S-4S 

Zl-9 

0 

CALC/TRER  + RPT 

2K-4K 

PWU-9 

I 

RRPK/TRER  + RPT 

1N-4N 

PYI-9 

I 

RRPK/TRER  + RPT 

2N-4N 

WU-9 

0 

CALC/TRER  + RPT 

IP-4P 

* Yl-9 

0 

CALC/TRER  + RPT 

2Q-4Q 

PSUCC1-9 

0 

CALC/TRER  + RPT 

2T-4T 

PFAIL1-9 

0 

CALC/TRER  + RPT 

2U-4U 

PI1-9 

I 

RRPK/TRER  + RPT 

2V-4V 

RKPKGI-9 

0 

CALC/TRER  + RPT 

2VJ-4VI 

C0ST1-9 

I 

RRPK/CMR  + RPT 

2X-4X 

DECO-9 

0 

CALC/TRER  + RPT 

IT,  1Z-4Z 

M-4M, 


in-  no 
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Program  P58  (ETAM  Variable  Scaling  Program). 

Purpose:  The  objective  of  this  program  is  to  develop  a system  utility  for  each 
Decision  Tree  outcome,  based  upon  a summation  of  the  weighted  value  of 
each  relevant  attribute  which  is  affected  by  the  introduction  of  the 
innovation  into  the  system.  The  assessor  will  be  provided  interactive 
capability  in  assigning  attribute  values.  Prompting  techniques  will 
improve  the  validity  of  the  overall  measure  of  utility  developed  for 
each  potential  decision  outcome.  The  first  step  in  the  process  will 
be  to  graphically  characterize  the  variables  associated  with  each  out- 
come state.  In  order  to  perform  this  task,  the  assessor  must  have  a 
sound  understanding  of  the  organizational  goals  into  which  the  proposed 
innovation  is  planned  to  be  introduced.  Following  the  graphic  charac- 
terization of  the  variables  and  outcome  states,  the  assessor  will 
select  either  the  path  which  treats  non-continuous  base  attributes  or 
the  path  which  handles  continuous  base  attributes.  A procedure  is 
presented  for  development  of  variable  values  under  each  of  these 
conditions.  Certain  consistency  checks  can  be  made  in  order  to  improve 
the  validity  of  the  utilities  which  the  assessor  derived. 

Inputs:  The  program  inputs  and  sources  are: 

1.  Terminal  displayed  options  which  the  assessor  selects,  namely: 

o Continuous  base  attribute 

o Non-continuous  base  attribute 

o Instructions 

o Quit 

2.  The  assessor  should  have  access  the  scenarios  stored  in  the  SCEN 
file  of  the  Project  Data  Base.  A printout  of  that  file  was 
selectable  under  Program  P60. 

3.  The  relevant  qualitative  variables  which  are  affected  by  intro- 
ducing the  innovation  into  the  system  were  stored  in  the  VQAL 
file  and  interactively  transferred  to  the  VARF  file  under  Program 
P60.  Thus  all  of  the  variables  associated  with  the  Decision  Tree 
outcome  states,  should  now  be  contained  in  the  VARF  file. 

4.  The  Cost/Savings  stream  over  the  planning  period  being  considered 
for  the  innovation  will  have  been  stored  in  the  CMR  file  as  a 
result  of  one  or  more  cost  model  runs.  A Record  Type  9 will  have 
summary  information  of  the  project  cost  and  savings. 

5.  User  interaction  based  upon  terminal  prompting  will  provide  the 
inputs  necessary  to  converge  on  the  system  utility  for  each  of 
the  outcome  states. 

Process:  The  major  program  process  steps  are: 

1.  The  Program  PbE  is  called  by  Program  P60  if  the  user  selects  the 
DEVELOP  OUTCOME  UTILITIES  (OPTION  6). 

2.  A display  of  the  select  options  is  presented  to  the  assessor. 

The  assessor  selects  either  the  continuous  base  attribute  option, 

or  the  non-continuous  base  attribute  option.  The  option  exists  also 
for  printing  instructions  in  which  case  a series  of  instructions  will 
be  printed  and  the  assessor  will  then  again  be  displayed  the  options 
for  another  selection.  The  assessor  can  also  QUIT,  in  which  case 
the  program  returns  to  the  major  select  options  under  Program  P60. 
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3.  Under  either  of  the  selected  options  the  assessor  will  prepare  a 
descriptive  (graphical)  representation  of  the  attributes. 

4.  This  step  more  or  less  extends  the  characterization  of  the  outcome 
states  as  outlined  in  the  SCEN  file.  Its  purpose  is  to  prompt  a 
continued  orientation  to  the  operational  meaning  inherent  in  the 
selected  relevant  attributes.  If  this  step  does  not  cause  the 
qualitative  variables  to  be  anchored  in  some  operational  context, 
then  in  all  probability  the  variables  should  be  eliminated  from 
consideration. 

It  is  important,  at  this  point,  that  the  assessor  have  a sound 
understanding  of  the  goals  of  the  organization  into  which  the 
proposed  innovation  would  be  introduced.  For  each  attribute 
impacted  by  the  innovation,  the  assessor  should  be  able  to  identify 
some  organizational  goal  to  which  it  can  be  related.  The  prior 
emphasis  upon  an  operational  and  organizational  context  for  each 
qualitative  attribute  should  now  be  apparent.  The  assessor  will 
attempt  to  describe  the  change  to  or  away  from  some  goal -state 
which  occurs  from  introducing  the  innovation.  Estimates  will 
then  be  made  for  the  states  of  each  of  the  other  potential  outcomes. 
Figure  1 1 1-46  provides  a sample  result  of  the  estimation  process. 
Attribute  \]  (On-Job  Flexibility)  is  estimated  at  its  present 
state  (x] ) which  is  the  state  assuming  the  innovation  is  rejected. 

The  assessor  believes  that  this  present  state  is  about  halfway 
(50%)  between  what  is  considered  barely  tolerable  and  what  is 
considered  ideal.  Next,  the  assessor  considers  the  effect  of  the 
innovation  on  this  variable,  if  the  innovation  is  successful 
(Outcome  A).  This  situation  is  estimated  to  improve  the  state  of 
the  On-Job  Flexibility  from  its  present  state  (50%)  to  a point 
about  halfway  to  the  ideal  state.  This  is  shown  as  x]  which  is 
at  the  75%  point.  Intermediate  outcomes  (B  and  C)  ar£  then 
estimated.  This  same  process  is  performed  for  the  remaining 
attributes  (x2,  x3  and  x**)- 

The  preceding  approach  called  for  direct  estimates  of  the  outcome 
points  by  attribute.  There  are  several  other  techniques  which 
may  be  applied  both  here,  and  in  a later  step  dealing  with  importance 
estimates.  One  or  more  of  these  alternate  methods  should  be 
considered  as  a test  for  consistency  (reliability)  of  the  values 
estimated. 

1)  Ratio  tests  over  range  xt  to  x9.  Considering  the  x,  attribute 
shown  in  Figure  III -46 , test  ti\e  reasonability  o^  tne  ratios 
of  each  outcome  level  to  the  goal  and  tolerable  states.  For 
example: 

a)  Is  the  present  state  (D)  just  as  far  away  from  the 
ideal  as  it  is  from  the  least  tolerable  state. 

b)  Does  satisfactory  Introduction  of  the  innovation  (Outcome 
A)  accomplish  50  percent  of  what  is  expected  in  order 

to  achieve  the  goal  state  (x^). 
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c)  Does  a failure  to  implement  (Outcome  C)  result  in 
essentially  no  change. 

d)  If  the  innovation  is  implemented  but  fails  to  improve 
the  system  as  anticipated,  (Outcome  B),  does  it  contribu 
50%  as  much  toward  accomplishment  of  the  goal  as  success 
introduction  (Outcome  A). 

2)  Ratio  tests  over  range  of  possible  outcomes.  This  is  similar 
to  the  preceding  test,  except  that  the  ratio  of  one  outcome 
to  another,  independent  of  its  relationship  to  the  extremes, 
is  checked.  For  example,  consider  the  outcomes  for  the 
attribute  x2  in  Figure  1 1 1-46 . 

a)  Does  successful  introduction  of  the  innovation  (Outcome 
A)  provide  as  much  satisfaction  or  utility,  as  the 
unsuccessful  introduction  (Outcome  B)  reflects  in 
decreased  satisfaction  or  utility. 

b)  Does  twice  as  much  decreased  satisfaction  or  utility 
result  from  unsuccessful  introduction  (Outcome  B)  as 
does  from  a failure  to  implement  (Outcome  C). 

3)  Indifference-lotteries  over  range  x.  to  x^.  The  indifference- 
lottery  method  look  for  the  point  at  which  the  assessor  is 
indifferent  between  accepting  the  results  of  a specific 
outcome  with  certainty  and  taking  a gamble  at  prescribed 

odds  with  the  potential  for  receiving  either  one  of  two 
outcomes.  For  example,  again  considering  attribute  x-j 
in  Figure  1 1 1-46 , would  you  as  the  assessor  be  indifferent 
to  maintaining  the  present  state  (Outcome  D)  with  certainty, 
as  opposed  to  taking  a gamble  with  50:50  odds  of  reaching 
the  ideal  state  (x^)  or  dropping  to  the  least  toVerable 
state  (xj-).  If  this  is  reasonable,  then  the  placement  of 
Outcome  D at  the  50%  level  appears  correct. 

The  preceding  sequence  of  estimates  will  be  prompted  for  via  the 
terminal.  The  assessor  will  input  the  values  requested  which  will 
be  temporarily  stored  until  the  completion  of  the  utility  scaling 
process.  At  that  time,  an  output  report  displaying  the  outcome 
states  by  scaled  variable  (Report  DT02)  will  be  printed. 

5.  If  there  exists  one  relatively  important  and  quantified  benefit 
variable,  such  as  dollars,  then  this  path  through  Steps  5 and  6 
will  be  followed.  Each  level  of  each  of  the  other  attributes  can 
be  converted  into  units  of  the  base  attribute  by  applying  a 
tradeoff  procedure.  The  graphical  representation  of  outcome 
states  by  attribute  developed  in  Step  4 and  presented  in  Figure 
1 1 1-46  will  be  used  as  the  basic  reference  for  this  conversion 
process. 


! 
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For  example,  in  Figure  1 1 1-46 , consider  that  the  dollars  (cost/ 
savings  variable  x^)  saved  as  a result  of  successfully  intro- 
ducing an  innovation  are  a dominant  aspect  of  the  decision-making 
process.  A specific  dollar  cost  or  savings  will  have  been  obtained 
as  a result  of  the  cost  model  processing  of  the  outcome  scenario 
parameters  previously  created.  Assume  Outcome  A produces  a savings 
of  $500,000  (500K)  and  Outcomes  C and  B result  in  losses  or  costs 
of  100K  and  200K,  respectively.  Outcome  D where  the  innovation 
has  been  rejected  is  neutral,  or  zero  cost/savings.  The  dollar 
variable,  then,  has  a relatively  wide  range  of  outcome  possibili- 
ties, and  is  also  significant  in  terms  of  the  values  taken  on 
across  the  potential  outcomes.  The  dollar  dimension  is  therefore 
selected  as  the  continuous  base  attribute  into  which  the  levels 
or  states  of  each  of  the  other  attributes  will  be  converted. 

Consider  the  attribute,  On-Job  Flexibility,  (x-j  in  Figure  III -46 ) . 
Outcome  A appears  to  have  a substantial  positive  value  in  that  it 
has  the  effect  of  moving  the  organization  half-way  to  its  ideal 
or  goal-state  (from  a present  state  of  50  percent,  to  75  percent). 
The  question  can  be  asked:  "How  many  dollars  of  the  organization's 

budget  would  the  decision  maker  be  willing  to  invest  to  avoid 
losing  about  the  same  level  of  On-Job  Flexibility  as  results  from 
successful  introduction  of  the  innovation?"  The  question  should 
be  answered  independent  of  the  range  of  cost/savings  possibilities 
related  to  the  innovation  decision.  If  the  answer  were  50K 
dollars,  then  this  represents  additional  value  (over  and  above 
the  500K  savings)  to  Outcome  A.  This  same  process  is  repeated 
for  the  remaining  states  of  variable  x-j , as  well  as  for  each 
state  of  the  variables  X£  and  X3.  Where  an  outcome  is  less  than 
the  neutral  state  (as  with  Outcomes  B and  C for  X2) * then  the 
appropriate  question  to  be  asked  by  the  assessor  would  be:  "How 

many  dollars  of  the  organization's  budget  would  a decision  maker 
be  willing  to  pay  to  avoid  the  decreased  level  of  the  attribute 
resulting  from  a decision  (and  potential  outcome  from  the  decision) 
regarding  the  innovation?"  The  results  of  such  a tradeoff  analysis 
is  tabulated,  for  example,  as  follows: 


ATTRIBUTE 


0 

A 

$(X4) 

+500K 

Skill (x3) 
+30K 

Motiv^) 
+1  OK 

On-Job  Flex(x^  ) 
+20K 

u 

T 

B 

-200K 

+20K 

-10K 

+1  OK 

C 

0 

C 

-100K 

+10K 

-5K 

0 

M 

E 

D 

0 

0 

0 

0 

Intermediary  verbal  descriptions  for  each  level  would  undoubtedly 
be  used  in  arriving  at  these  types  of  value  estimates  (e.g., 
significant  increase  in  skill  level).  These  should  come  about 
through  the  analysis  from  Step  4. 
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The  output  of  this  step  will  be  defined  in  the  output  section  of 
this  program  and  will  be  displayed  to  the  user  in  the  report  on 
variable  values  by  outcome  (DT04). 

6.  As  a result  of  the  previous  step,  each  level  or  state  of  all 
rel event  attributes  was  converted  into  a value  reflected  in  the 
same  units  as  those  of  a selected  important  continuous  base 
variable  such  as  dollars.  If  the  organization  or  system  utility 
for  units  of  the  base  dimension  is  linear,  i.e.,  200K  dollars  is 
preferred  exactly  twice  as  much  as  100K  dollars,  then  the  values 
identified  for  each  outcome  state  can  be  summed  across  Outcome  A, 
then  Outcome  B,  etc.  For  example,  the  value  of  Outcome  A would 
be  the  sum  of  500K  dollars,  30K  dollars  worth  of  skill  level,  10K 
dollars  worth  of  motivation,  and  20K  dollars  worth  of  on- job 
flexibility;  it  would  be  560K  dollars. 

In  many  cases  the  organization  or  system  utility  for  dollars  will 
not  be  linear.  The  initial  100K  dollar  savings  may  be  worth  much 
more  than  another  100K  (200K  total)  because  it  has  greater  utility 
in  any  application  the  organization  has  for  dollars.  This  should 
be  clear  if  the  first  100K  could  be  invested  in  a machine  that 
would  double  capacity,  increase  profits,  and  produce  a 100  percent 
return-on-investment;  while  the  second  TOOK  has  a utility  ten 
times  that  of  the  second  100K.  This  may  be  an  extreme  example, 
but  it  should  make  the  point. 

The  problem,  therefore,  in  this  step  is  to  insure  that  the  utility 
ultimately  determined  at  some  particular  outcome  reflects  the  sum 
of  the  utilities  for  each  of  the  attributes  estimated  at  this 
state.  An  important  assumption  in  this  process  is  that  when  a 
tradeoff  estimate  is  made  for  some  attribute  state  in  units  of 
the  continuous  base  variable,  not  in  terms  of  its  actual  value. 

For  example,  when  the  system  through  the  action  of  the  decision 
maker  is  willing  to  commit  (as  is  expected  when  the  tradeoff 
value  is  decided  upon)  30K  dollars  to  achieve  some  X units  worth 
of  Increased  skill  level,,  that  committed  amount  takes  into  account 
the  utility  that  30K  dollars  has  to  the  system.  Therefore  the 
10K  dollars  estimated  for  an  increased  level  of  motivation  woula 
have  one-third  the  utility  of  the  30K  only  if  the  system  preference 
for  each  1 OK  increment  were  equal.  Thus,  before  the  utility  of 
an  outcome  can  be  determined,  a utility  function  must  be  assessed 
across  values  of  the  base  attributes.  Procedural ly  this  can  be 
accomplished  as  follows: 

1)  Specify  the  most  and  least  desirable  states  of  the  base 
attribute.  For  example,  the  most  desirable  may  be  +5tjk 
dollars  while  the  least  desirable  may  be  -200K  dollars.  Let 
these  states  be  assigned  arbitrary  utilities  of  100  and  0, 
respectively. 

2)  Select  intermediate  points  and  using  a direct  estimation, 
ratio;  and/or  lottery  technique,  determine  their  utility  on 
the  0 to  100  scale.  These  techniques  were  presented  in  Step 
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3)  Graph  the  utility  function  for  the  base  variable  in  order 
that  states  cf  the  other  attributes  can  be  transformed  into 
a corresponding  utility  figure  through  interpolation.  Once 
the  utility  function  has  been  graphed,  it  seems  more  psycho- 
logically sound  to  transform  the  utility  scale  by  making 
zero  utility  equal  zero  units  of  the  base  variable,  thus 
providing  both  positive  and  negative  values  for  utility. 


Figure  1 1 1-47  shows  an  example  of  a graph  for  the  utility  of 
dollars.  Some  of  the  interpretations  made  from  it  are  as  follows. 
They  also  represent  samples  of  the  means  by  which  the  graph  could 
have  originally  been  constructed. 

a.  There  is  as  much  utility  to  the  system  in  obtaining  5Q0K 
dollars  as  there  is  disutility  in  losing  200K  dollars. 

b.  The  system  represented  by  the  decision  maker  is  relatively 
indifferent  to  a 50:50  gamble  between  winning  500K  dollars 
and  losing  - 20CK  dollars.  The  decision  maker  is,  therefore, 
conservative  and  exhibits  a substantial  aversion  toward  risk 
since  a risk  neutral  decision  maker  would  accept  the  same 
gamble  when  the  chances  of  winning  the  500K  dollars  are  only 
about  30  percent.  ( ( p( 500)  + ( 1 -p) ( -200) ) p = 28.6%). 

c.  There  is  as  much  utility  to  the  system  in  receiving  the  100K 
dollar  increment  from  200K  to  300K  as  there  is  in  receiving 
the  200K  dollar  increment  from  300K  to  500K.  In  other  ^ords 
there  appears  to  be  diminishing  marginal  utility  for  each 
additional  100K  dollar  increment. 

d.  There  is  four  times  the  disutility  from  losing  the  first  100K 
dollars  as  there  is  in  losing  the  second  100K  dollar  increment. 
(Perhaps  losing  100K  is  so  damaging  that  the  additional  100K 
cannot  do  that  much  more  harm.) 

The  states  of  a 1 .1  attributes  previously  converted  into  units  of 
the  base  dimension  can  now  be  translated  directly  into  utilities 
by  interpolating  from  the  graph.  The  following  example  taken 
from  Step  6 shows  the  results.  Note  that  the  utility  scale 
having  both  positive  and  negative  units  is  used.  The  original 
dollar  values  are  shown  in  parentheses. 


ATTRIBUTE 


$(X4 ) 

Skill (x3) 

Moti v(x2) 

On-Job  Flex(x-j 

50  (500K) 

8 (30K) 

2 ( 1 OK)  Est. 

5 (20K)  Est 

-50  (-200K) 

5 (20K)  Est. 

-8  ( -1  OK) 

2 (10K)  Est 

-40  (-100K) 

2 (10K)  Est. 

-5  ( -5K)  Est. 

0 (OK) 

0 (OK) 

0 (OK) 

0 (OK) 

0 (OK) 
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The  total  utility  for  each  outcome  can  be  determined  by  adding 
across:  Outcome  A = 65,  Outcome  B = -51 , Outcome  C = -43,  and 

Outcome  D = 0. 

An  output  report  giving  the  variable  values  by  outcome  (DT04) 
will  be  produced  as  an  output  from  this  step. 

7.  The  following  technique  will  be  applied  when  it  has  been  determined 
that  no  dominant  continuous  base  attribute  exists.  This  and  the 
following  Step  8 will  permit  the  development  of  a system  utility 
figure  for  each  decision  tree  outcome;  thereby  parelleling  the 
objective  of  Steps  5 and  6 where  there  exists  a dominant  base 
variable. 

1)  Consider  the  goal-related  operational  effects  projected  to 
be  achieved  from  successful  introduction  of  the  innovation 
(points  A-j , A^,  A^>  and  in  Figure  1 1 1-46). 

2)  Select  the  most  important  effect  considering  the  importance 
of  the  system  goal  (all  points  X9  in  Figure  1 1 1-46),  the 
present  status  of  the  attribute, 1 and  the  size  of  the  incremental 
change  resulting  from  successfully  introducing  the  innovation 
into  the  system. 

3)  Select  the  least  important  effect  considering  the  same 
factors  as  In  2)  above.  These  are  the  points  designated  as 
Xt  Figure  III-46. 

4)  Rank  all  other  attributes  considering  the  same  factors  as  ir 

2). 

5)  Assign  the  most  important  attribute  selected  in  2)  a value 

of  1.0.  Directly  estimate  the  importance  of  other  attributes 
in  relation  to  the  most  important  one. 

6)  Check  the  consistency  of  the  importance  weights  assigned  by 
the  direct  estimate  technique  by  performing  one  or  more  of 
the  following  additional  techniques  for  estimating  importance. 

a)  Assess  ratios  of  importance  between  all  combinations. 

b)  Assess  points  of  indifference  using  a lottery  technique. 

Returning  to  the  example  presented  in  Figure  1 1 1-46,  assume  that 
the  continuous  base  variable  costs/savings  (x^)  is  relatively 
unimportant,  and  that  it  is  felt  more  appropriate  to  use  the  pro- 
cedures in  Steps  7 and  8 for  assessing  the  system  utility  at  each 
outcome.  The  following  sequence  of  thought-processes  might  illus- 
trate the  technique  for  arriving  at  relative  importance  values  for 
the  four  attributes  of  Figure  111-46. 
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a.  The  most  important  attribute  is  Skill  Level  (x^)  since  it  is 
an  important  system  goal  and  its  present  state Jis  relatively 
low. 

b.  Motivation  Is  least  Important  since  it  is  presently  at  a 
fairly  high  level . 

c.  On-Job  Flexibility  is  considered  more  important  than  Cost/ 
Savings  because  it  contributes  significantly  to  overall 
mission  in  terms  of  preparedness. 

d.  Skill  Level  is  assigned  a value  of  1.0;  the  remainder  in 
order  are  On-Job  Flexibility,  Cost/Savings,  then  Motivation. 

e.  The  Motivation  attribute  is  less  than  half  as  important  as 
Skill  Level;  it  is  therefore  assigned  an  initial  value  of  .4. 

f.  On-Job  Flexibility  is  nearly  as  Important  as  Skill  Level. 
Motivation  is  about  half  as  important  as  On-Job  Flexibility, 
therefore  On-Job  Flexibility  is  assigned  a value  of  .8. 

g.  The  Cost/Savings  variable  is  slightly  more  important  than 
Motivation,  and  is  judged  about  half  as  important  as  Skill 
Level;  it  is,  therefore  assigned  a value  of  .5. 

The  final  importance  weights  to  be  assigned  each  of  the  four  attri- 
butes will  be: 


Skill  Level 

- 1.0 

On-Job  Flexibility 

- .8 

Cost/Savings 

- .5 

Motivation 

- .4 

These  will  be  applied  to  the  scales  shown  by  the  example  in  Figure 
1 1 1-46  to  develop  a system  utility  by  outcome. 

At  the  conclusion  of  Step  7 each  benefit  variable  has  been  quanti- 
fied In  terms  of  contribution  to  some  specific  goal,  and  its 
weight  of  relative  Importance  to  the  overall  system  mission  has 
been  estimated.  In  this  step,  the  relative  importance  weighting 
factor  will  be  applied  to  adjust  the  lengths  for  each  attribute; 
the  result  will  be  a corresponding  assignment  of  system  uti  lity 
to  each  outcome  of  the  rescaled  attributes.  The  following  tech- 
nique Is  applied  to  determine  the  utility  for  each  outcome. 

Figure  1 1 1-48  presents  an  example  of  the  results. 

1)  Assign  an  arbitrary  scale  for  system  utilities  with  zero  as 
the  midpoint,  the  midpoint  representing  the  present  system 
state. 

2)  Select  the  scaled  attribute  weighted  as  most  important  and 
anchor  its  present  state  (point  D)  at  the  zero  point  of  the 
arbitrary  scale. 


FIGURE  111-48.  APPLICATION  OF  IMPORTANCE  WEIGHTS  TO 

DEVELOP  OVERALL  SYSTEM  UTILITY  BY  OUTCOME 
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3)  Let  the  total  length  of  the  most  important  attribute  scan 
exactly  50  percent  of  the  arbitrary  system  utilities  scale 
(this  is  for  convenience  and  will  permit  all  other  attributes 
of  lesser  importance  to  be  contained  within  the  scale). 

Skill  Level  (x^)  in  Figure  III -48  is  shown  to  extend  from 
zero  to  plus  fifty  since  its  anchor  point  (D)  was  at  the 
left  most  point  on  its  scale  in  Figure  III -46 . 


4)  Select  the  next  most  Important  attribute  and  anchor  its 
present  state  (point  D)  to  the  zero  point  of  the  arbitrary 
scale. 

5)  Let  the  total  length  be  proportional  to  the  length  of  the 
most  important  attribute  based  upon  its  relative  importance 
weight  as  determined  In  Step  7.  In  Figure  III-48,  On-Job 
Flexibility  with  a relative  Importance  of  .8  Is  shown  to 
extend  a length  of  forty  system  utility  units  (.8  x 50). 

Since  its  anchor  point  (D)  Is  also  at  the  left  most  point  on 
its  scale  from  Figure  III -46 , it  is  shown  to  extend  from 
zero  to  plus  forty  in  Figure  III -48 . 

6)  Repeat  4)  and  5)  above  for  the  remaining  attributes.  Figure 
1 1 1-46  shows  that  the  other  two  attributes,  Cost/Savings  and 
Motivation,  extend  over  twenty-five  (.5  x 50)  and  twenty  (.4 
x 50)  units,  respectively.  Since  their  anchor  points  (D) 
are  between  other  outcome  points,  their  scales  extend  in 
both  a positive  and  negative  direction  from  the  zero  or 
midpoint. 

7)  Read  the  system  utilities  for  all  A outcomes  from  the  scale 

and  add  them  algebraically,  Thus  from  Figure  III -48 , = io, 

x]  * 17,  x]  * 40,  and  xl  * 50;  the  total  of  all  A outcomes 

i?  117.  1 J 


8)  Repeat  7)  above  for  Outcomes  B,  C and  D.  Outcome  D will 
always  be  zero.  Thus  from  Figure  1 1 1-48,  x£  = -10,  x? 
xy  * 20,  and  x^„*  34;  Outcome  B total  is  36.  Outcome  C 


= -8, 


tdtals  8 with  « 


■5,  xi  = -4.  x 


0 and 


x3  = 
x3 


17. 


In  summary,  then,  the  total  of  the  relevant  attributes  across 
each  major  outcome  is  as  follows: 


ATTRIBUTE 

OUTCOME  A 

OUTCOME  B 

OUTCOME  C 

OUTCOME  D 

Mot1vat1on(x2) 

10 

-10 

-5 

0 

Cost/Sav1ngs(x4) 

17 

-8 

-4 

0 

On-Job  Flexl- 
bll 1ty(x] ) 

40 

20 

0 

0 

Skill  Level (x3) 

50 

34 

V7 

_0 

Total 

117 

36 

8 

0 
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These  are  the  utility  figures  which  will  be  inserted  into  the 
decision  tree  after  a consistency  check  has  been  performed  in 
Step  9. 

As  a result  of  this  step  an  output  report  giving  the  variable 
values  by  outcome  (DT04)  will  be  generated  for  the  assessor  to 
review. 

At  this  point  in  the  procedure,  some  system  utility  for  each 
decision  tree  outcome  will  have  been  generated,  either  as  a 
result  of  the  conversion  of  relevant  variables  into  units  of  a 
continuous  base  variable  as  was  performed  in  Steps  5 and  6,  or 
through  application  of  the  techniques  outlined  in  Steps  7 and  8. 
Both  Steps  6 and  8 recommended  crosschecks  for  the  consistency  of 
the  subjective  estimates.  In  this  step,  a wholistic  common  sense 
appraisal  of  the  relative  total  utility  of  each  outcome  is  pro- 
posed. The  consistency  check  steps  are  as  follows: 

1)  Check  the  ranking  of  the  outcomes  for  reasonability.  From 
the  examples  summarized  at  the  end  of  Steps  6 and  8,  the 
following  total  utilities  by  outcome  were  obtained. 


Step  6 Step  8 


OUTCOME 

A 

65 

117 

OUTCOME 

B 

-51 

36 

OUTCOME 

C 

43 

8 

OUTCOME 

D 

1 

0 

As  is  seen  the  results  are  quite  different  between  the  two, 
however,  the  assumptions  were  quite  different  and  there  was 
no  intent  that  the  results  in  any  way  be  similar. 

At  this  stage,  the  assessor  should  make  the  intuitive  assess- 
ment on  the  order  of  outcomes  for  each  of  the  innovations 
leading  to  the  above  summarized  results. 

a)  Is  the  order  reasonable  for  the  innovation  processed 
through  Step  6,  i.e.,  A,  D,  C,  then  B? 

b)  Is  the  order  reasonable  for  the  innovation  processed 
through  Step  8,  i.e.,  A,  B,  C,  then  D?  Does  it  seem 
reasonable  that  acceptance  of  the  innovation,  no  matter 
what  the  final  outcome,  will  provide  more  utility  to 
the  system  than  outright  rejection? 

2)  Check  the  ratios  of  outcomes  to  one  another  for  reasonableness. 

a)  From  the  Step  6 results,  does  Outcome  B appear  to 

provide  as  much  negative  utility  as  Outcome  A provides 
positive  utility?  Are  Outcomes  B and  C just  about  as 
bad  for  the  system? 


TAEG  REPORT  NO.  40 


b)  From  the  Step  8 results,  does  Outcome  A appear  to 
provide  over  three  times  the  utility  as  Outcome  B? 

3)  If  the  results  of  1)  and  2)  above  do  not  appear  to  be  reason- 
able In  terms  of  the  separations  of  the  values  of  the  outcome 
and  their  negative  or  positive  direction  with  respect  to  the 
present  system  state,  reexamine  the  following. 

a)  Have  all  relevant  variables  (impacts  of  the  innovation) 
been  explicitly  Identified  and/or  taken  into  account? 

b)  Have  realistic  scenarios  been  created  for  each  outcome 
considering  operational  effects? 

c)  Have  the  individual  attributes  been  realistically 
assessed  with  respect  to  the  system's  goals? 

d)  Have  the  attributes  been  weighted  realistically? 

Outputs:  The  major  program  outputs  and  their  destination  are: 

1)  The  final  list  of  relevant  variables  to  which  values  will  be 
assigned.  These  will  be  selected  from  the  list  originally  stored 
In  the  VQAL  project  file,  and  along  with  the  Cost/Savings  vari- 
able, will  be  stored  In  the  VARF  project  file. 

2)  The  relative  importance  of  each  of  the  relevant  variables.  An 
Importance  weighting  factor  from  .1  to  1.0  will  be  assigned  to 
each  variable  and  stored  In  the  VARF  project  file  In  the  data 
record  for  the  variable. 

3)  The  weighted  value  of  each  relevant  variable  (including  Cost/ 
Savings)  for  each  of  the  potential  decision  outcomes.  These 
values  will  be  developed  by  the  assessor  and  stored  in  the  VARF 
project  file. 

4)  A total  of  the  weighted  values  of  each  relevant  variable  for  each 
potential  decision  outcome.  These  will  be  calculated  by  the 
program  and  stored  as  a separate  data  record  (999  record)  in  the 
VARF  project  fl1*. 

5)  The  graphical  characterization  of  the  outcome  states  by  scaled 
variable  will  be  presented  In  an  ETAM  DECISION  OUTCOME  STATES  BY 
SCALED  VARIABLE  (DT02)  report.  The  format  of  this  report  Is 
shown  In  Figure  1 1 1-49.  The  SEQ  NO  (Gl-Zl)  and  VARIABLE  NAMF 
(G2-Z2)  columns  are  taken  from  the  VARF  file.  REL  IMP  (G3-Z3) 
would  represent  an  Initial  relative  Importance  loaded  into  the 
VARF  file  since  It  Is  not  developed  Interactively  until  process 
Step  7.  Since  It  Is  stored  In  the  VARF  file  as  a number  from  1 
to  10,  It  must  be  scaled  by  a factor  of  1/1 Oth  before  entering  on 
the  DT02  report.  The  PERCENT  OF  DISTANCE  TO  GOAL  STATE  (G4-Z4) 

Is  developed  Interactively  In  process  Step  4.  An  example  of  a 
DT02  report  Is  shown  In  Figure  111-50. 
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FIGURE  111-49.  FORMAT  OF  ETAM  DECISION  OUTCOME  STATES  BY 
SCALED  VARIABLE  REPORT  (DT02) 
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FIGURE  111-50.  EXAMPLE  OF  ETAM  DECISION  OUTCOME  STATES  BY 
SCALED  VARIABLE  REPORT  (DT02) 
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6)  The  ETAM  DECISION  OUTCOME  STATES  BY  VARIABLE  (DT03)  report 
presents  the  variables  as  scaled  in  Step  8 to  represent  System 
Utilities.  Columns  1,  2,  and  3 are  the  same  as  defined  in  the 
DT03  report.  Column  4 portrays  the  variables  with  the  Reject 
outcome  for  each  variable  set  to  the  zero  point  on  the  scale,  and 
the  remaining  outcomes  scaled  by  the  relative  importance  factor. 

A SEQ  NO  999  (Zl)  record  summarizes  the  system  utilities  by 
outcome  (Z4).  Figure  1 1 1-51  shows  the  format  of  report  DT03  and 
Figure  1 1 1 - 52  provides  an  example. 

7)  Another  summary  report  gives  the  ETAM  DECISION  VARIABLE  VALUES 
BY  OUTCOME  (DT04).  Figure  1 1 1 - 5 3 shows  the  DT04  report  format. 
Columns  1,2,  and  3 are  the  same  as  reports  DT02  and  DT03. 

Columns  4,  5,  6,  and  7 give  the  quantification  of  the  outcomes  by 
variable  as  developed  interactively  either  in  Step  6 for  continu- 
ous base  attributes,  or  in  Step  8 for  non-continuous  base  attri- 
butes. Figures  1 1 1 -54  and  1 1 1-55  give  examples  of  the  DT04 
report  for  the  cases  when  Steps  5-6  were  taken  or  Steps  7-8  were 
followed,  respectively. 

Program  P59  (Assigning  Probabilities) 

Purpose:  The  decision  tree  structure  incorporated  within  ETAM  requires  the 

assessor  estimate  a number  of  probabilities  associated  with  various 
decision  outcomes.  This  porgram  interactively  prompts  the  user  in 
assigning  probability  values  to  any  of  the  decision  tree  nodes  which 
were  shown  in  Figure  1 1 1 - 44 . The  probabilities  on  the  path  with 
NO  RISK  REDUCTION  PROJECTS  (DECO)  are  estimated  based  upon  the  risks 
which  were  outlined  in  the  RKP  file  (Figure  1 1 1 -1 0 . ) . These  risks  can 
be  associated  wither  with  getting  the  innovation  physically  implemented; 
i.e.  designed,  developed,  manufactured,  and  installed,  or  with  gaining 
user  acceptance  once  the  innovation  is  installed  and  operational.  Thus, 
the  assessor  in  performing  this  probability  assignment  function  must 
have  a clear  concept  of  the  implications  of  the  identified  risks  upon 
the  incorporation  of  the  innovation  into  the  system. 

Probabilities  may  be  estimated  initially  prior  to  creating  the  Project 
Data  Base.  This  program  may  also  be  called  as  a part  of  the  ETAM  Risk 
Reduction  Project  Grouping  (Program  P56),  or  it  may  be  used  within  the 
ETAM  Decision  Analysis  function  to  refine  the  probabilities  prior  to 
running  the  program  to  calculate  the  decision  variables.  In  all  cases, 
the  probabilities  are  stored  in  the  RRPK  file.  Its  format  was  previously 
described  within  the  section  on  the  Project  Data  Base,  and  an  example  of 
its  contents  were  shown  in  Figure  III-12. 

In  addition  to  estimating  probabilities  for  for  NO  RISK  PKG  path,  proba- 
bilities are  required  for  each  path  for  which  a group  of  risk  reduction 
packages  has  been  defined. 

Inputs:  This  program  has  the  capability  of  permitting  the  assessor  to  print  the 

RPK,  RRPJ,  and  RRPK  files  via  terminal  control.  Interactive  terminal 
prompting  elicits  user  responses  which,  hopefully,  converge  cn  a set  of 
valid  probability  estimates  for  the  nodes  of  the  decision  tree.  The 
potential  variables  to  be  estimated  are:  PX,  PW,  PX1  thru  PX9,  PW1  thru 

PW9,  PZ1  thru  PZ9,  and  PI1  thru  PI9. 
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Process:  The  major  program  process  steps  are: 

1.  Provide  user  with  option  to  review  RRPK  file  contents. 

2.  Permit  selection  of  Risk  Group  (0-9)  to  be  assigned  probabilities. 

3.  Permit  selection  of  specific  probability  groups  to  be  estimated. 

Group  1 - Probability  of  Risk  Package  Success 

These  probabilities  would  be  assigned  to  the  first 
three  of  the  nine  probabilities  in  the  1-1,  2-1,  3-1, 
etc.  records  of  the  RRPK  file  (the  0-1  record  does  not 
have  any  Risk  Package).  The  first  of  the  three  proba- 
bilities represents  the  most  pessimistic  estimate,  the 
middle  probability  the  median  estimate,  and  the  third 
probability  the  most  optimistic  estimate. 

Group  2 - Probability  of  Implementation  Success 

These  probabilities  would  be  assigned  to  the  second 
three  of  the  nine  probabilities  in  the  0-1,  1-1,  2-1, 

3-1,  etc.  records  of  the  RRPK  file.  The  three  proba- 
bilities are  in  the  same  order  as  Group  1. 

Group  3 - Probability  of  User  Acceptance  Success 

These  probabilities  are  assigned  to  the  third  three 
of  the  nine  probabilities  in  the  0-1.  1-1,  2-1,  3-1, 
etc.  records  of  the  RRPK  file.  The  three  probabilities 
are  in  the  same  order  as  Group  1. 

4.  Estimate  the  probability  selected.  This  is  an  interactive  session 
which  prompts  the  assessor  until  a reasonably  valid  estimate  of  the 
probability  group  under  consideration  has  been  made.  The  prompting 
sequence  would  start:* 

ESTIMATE  MOST  OPTIMISTIC  PROBABILITY  OF  (A)  SUCCESS??? 

(A)  depends  on  group  chosen,  i.e., 

Group  1 - (A)  = RISK  PKG 
Group  2 - (A)  = IMPLEMENTATION 
Group  3 - (A)  = USER  ACCEPTANCE 
User  responds:  80  for  example. 

ESTIMATE  MOST  PESSIMISTIC  PROBABILITY  OF  (A)  SUCCESS??? 

User  responds:  50  for  example. 

The  program  then  sets  up  a series  of  three  values  and  the  user  is 
requested  to  estimate  his  preference  for  the  range  into  which  the 
most  likely  probability  falls.  The  two  extreme  values  in  this 
prompting  sequence  will  be  the  prior  OPTIMISTIC/PESSIMISTIC  estimates. 
For  example: 

50  65  80 


♦This  technique  is  based  on  the  following  reference.  With  the  permission  of  Dr. 
Spetzler,  a section  of  the  referenced  report  is  reproduced  in  Appendix  E. 
Spetzler,  Carl  S.  and  Stael  von  Holstein,  Carl -Axel  S.,  1972.  Probability 
Encoding  in  Decision  Analysis,  paper  presented  at  the  0RSA-TIMS-AIEE  1972  Joint 
National  Meeting,  Atlantic  City,  N.  J.,  8-10  November  1972.  Stanford  Research 
Institute,  Menlo  Park,  California. 
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If  the  user  feels  the  most  likely  probability  Is  In  the  50  to  65 
range  rather  than  the  65  to  80  range,  the  user  responds  1_2,  l.e., 
range  1 Is  preferred  to  range  2.  Based  upon  this  response,  the 
user  Is  presented  with  a new  set  of  values,  for  example: 

50  60  80 

If  the  user  responds,  21;,  l.e.,  range  2 Is  preferred  to  range  1, 
another  set  of  numbers  would  be  printed. 

50  63  80 

This  would  continue  until  the  user  responds  EQ;  l.e.,  the  ranges 
are  equal  and  63  represents  the  most  likely  value. 

The  middle  value  Is  calculated  as:  If  the  preference  continues 

Initially  In  the  same  direction,  the  first  values  are  1 /6th  of 
the  difference  between  the  optimistic  and  pessimistic  ( (80-50 )/6 
*5).  If  the  preference  reverses,  then  the  reversed  value  Is  % 
of  the  value  used  In  the  previous  sequence.  A sample  sequence  Is 
as  follows: 


20 

50  • 

80 

21 

20 

60 

'80 

rr 

20 

70 

80 

T7 

20 

65 

80 

T? 

20 

60 

80 

rr 

20 

63 

80 

The  Interval  between  the  optimistic  and  pessimistic  estimates  is 
then  trisected  with  the  Initial  values  dependent  upon  the  results 
of  the  previous  bisected  sequence.  For  example,  the  Initial 
values  based  upon  a most  likely  value  of  63  would  be: 

20  41  71  80 

The  value  41  Is  halfway  between  the  pessimistic  value  of  20  and 
the  previously  selected  most  likely  value  of  63.  Likewise,  71  is 
halfway  between  the  most  likely  value  of  63  and  the  optimistic 
value  of  80. 

The  user  Is  requested  to  order  a preference  for  the  three  ranges; 
e.g.,  321 , range  3 (71-80)  Is  preferred  to  range  2 (41-71),  and 
range  2 is  preferred  to  range  1 (20-41).  The  user  continues  to 
receive  a presentation  of  ranges  until  a response  1«2«3  can  be 
made.  When  two  of  the  ranges  are  equal  and  preferred  to  the 
third.  It  Is  Indicated  as  1 ■ 23;  l.e.  range  1 and  2 are  equal 
and  preferred  to  range  3.  A sample  sequence  Is  as  follows: 


20 

41 

71 

80 

213 

20 

41 

67 

80 

m 

20 

41 

65 

80 

m 

20 

52 

65 

80 

TO 

20 

46 

64 

80 

20 

49 

63 

80 

20 

50 

64 

80 

w 

1 
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The  algorithm  for  successive  adjustments  is  similar  to  the 
bisection  algorithm,  that  is  initial  movements  in  one  direction 
are  % the  difference  that  exists  between  the  number  being  changed 
and  the  prior  most  likely  estimate;  e.g.,  71-63  = 8,  8/2  = 4, 

71-4  = 67.  This  half  adjusting  continues  until  a change  in  the 
preference  order  is  reflected  in  the  user  response,  after  which 
the  other  intermediate  value;  e.g.  41  is  half  adjusted.  63-41  = 22, 
22/2  = 11,  63-11  = 52. 

This  series  of  prompting  can  continue  as  shown  by  the  example  in 
Appendix  E.  Resolution  of  inconsistencies  between  the  results  of 
the  several  estimation  paths  should  be  reconciled  and  the  most 
likely  value  selected  for  insertion. 

5.  Repeat  Steps  4 and  5 until  all  probabilities  have  been  estimated. 

6.  The  values  selected  will  be  outputted  to  the  RRPK  file  opposite  the 
the  NO  RISK  PKG  (0-1),  RISK  PKG  1 (1-1),  etc.  records  following 
the  final  iteration  of  Steps  4 and  5. 

Outputs:  The  major  ouputs  from  this  program  are: 

1.  Displays  to  the  user  requesting  selection  of  options. 

o Assign  Probabilities 
o Instructions 

o Quit 

2.  Prompting  messages  requesting  preference  orders  for  probabi  1 ity 
ranges. 

3.  Storage  of  the  selected  probability  values  in  the  RRPK  file. 

Program  P60  (Calculate  Decision  Variables). 

Purpose:  This  program  calculates  the  decision  variables  for  each  alternative 

decision  path.  It  provides  an  ETAM  DECISION  ANALYSIS  DETAIL  REPORT 
DT01 . Program  P60  is  also  the  control  program  for  the  Decision 
Analysis  function.  It  checks  for  available  data  to  perform  a decision 
tree  calculation  then  provides  the  user  with  various  select  options 
for  renewing  Project  Data  Base  files  establishing  qualitative  variables, 
developing  outcome  utilities,  assigning  probabilities,  or  performing 
the  decision  tree  calculations.  The  primary  functions  were  shown  in 
the  previous  Figure  III -43 . 

Inputs:  The  major  inputs  to  this  program  are: 

1.  VARF  and  RRPK  data  for  determining  if  minimum  data  requirements 
are  met  for  performing  decision  tree  calculations. 

2.  User  terminal  selected  options  for  controlling  the  processing 
sequence. 

3.  VARF  and  CMR  file  date  for  performing  the  decision  tree 
calculations. 
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Process:  The  major  process  steps  are: 

1.  Check  RRPK  file  for  existence  of  Record  No.  0 which  would 
Indicate  NO  RISK  PKG  data  are  available  for  calculations. 

2.  Check  VARF  file  for  existence  of  Record  No.  999  which  would 
Indicate  a total  record  for  the  variable  values  stored  In 
that  file. 

3.  Set  a RUN  or  NORUN  condition  flag  based  upon  the  availability 
of  data  In  Steps  1 and  2. 

4.  The  user  Is  provided  a display  of  the  selection  options 
which  are: 

o Review  SCEN  file 

o Review  VQAL  file 

o Review  VARF  file 

o Establish  qualitative  variables  pattern 

o Run  tree  calculations 

o Develop  outcome  utilities 

o Assign  probabilities 

o Quit 

5.  The  user  selects  one  of  these  options  from  the  terminal  which 
will  cause  the  branch  either  to  program  PI  to  display  the  contents 
the  VARF,  VQAL,  or  or  SCEN  files;  or  to  Program  P58  to  develop 
outcome  utlltles,  or  to  Program  P59  to  assign  probabilities,  or 

to  sections  of  P60  for  either  performing  Decision  Tree  calculations 
or  establishing  the  yarlables  In  the  VQAL  file. 

6.  Provided  the  data  In  RRPK  and  PARF  was  sufficient  to  run,  the 
selection  of  the  RUNDTREE  calculations  options  will  perform  the 
calculations  and  print  the  DT01  report  as  previously  outlined 
In  the  section  entitled  ETAM  Dylslon  Analysis  Detail  Report 
(DTP! ).  The  processing  logic  for  these  calculations  Is  embodied 
In  that  section. 

7.  If  the  option  to  establish  qualitative  variables  Is  selected 
the  user  will  receive  a display  of  the  VQAL  variables  and  be 
requested  to  select  one  of  the  variables  for  entry  Into  the 
VARF  file.  Movement  of  that  variable  Into  the  VARF  file  will 
permit  that  variable  to  be  considered  In  the  assessment  of  outcome 
utilities  within  Program  P58. 

8.  Following  completion  of  any  of  the  options  other  than  Quit, 
the  user  will  be  branched  back  to  the  display  of  the  options 
for  selection  of  another  one. 

9.  The  Decision  Analysis  function  is  exited  by  entering  Quit  when 
the  select  options  are  displayed. 

Outputs:  The  major  outputs  of  Program  P60  are: 

1.  An  ETAM  Decision  Analysis  Detail  Report  (DT01)  will  result 
from  the  Decision  Tree  calculations  program. 

2.  The  output  of  the  Decision  Tree  calculations  will  be  placed 
In  the  TRER  file  for  future  reference. 

3.  If  the  selected  alternative  contains  a Risk  Reduction  package; 
l.e. , the  path  1,  2,  ....  9 Is  selected.  The  value  of  the 
Risk  Reduction  package;  l.e.,  C0ST1,  C0ST2,  etc.  Is  stored 

In  the  CMR  file  Project  Total  Record  NO.  9. 
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ETAM  FINANCIAL  ANALYSIS.  Financial  reports  may  be  generated  by  invoking  the  FIN 
routine.  A Project  Number  is  requested  which  initiates  a search  of  the  Alternate 
Project  File  for  that  project.  A financial  comparison  of  any  two  projects,  one 
of  which  must  include  the  primary  project,  may  be  run  based  upon  the  latest  Cost 
Model  Results  for  those  two  projects.  Financial  Analysis  report  FA01  will 
always  be  generated  for  the  financial  comparison  requested.  If  a financial 
analysis  of  the  primary  project  and  all  valid  alternate  projects  is  requested, 
an  FA01  report  will  be  generated  for  each,  and  in  addition,  an  FA02  summary 
report  will  also  be  generated.  Figure  1 1 1-56  shows  the  process  flow  for  the 
Financial  Analysis  routines  within  the  ETAM  Assessment  function. 

The  major  objectives  in  performing  the  financial  analysis  are  to: 

1.  Develop  a figure  of  merit  (rate  of  return  on  investment)  as  well  as 
other  comparative  factors  for  the  proposed  innovation  and  for  any 
potential  alternatives  to  the  proposed  innovation. 

2.  Develop  a rate  of  return  figure  for  the  incremental  investment  re- 
quired of  one  alternative  over  another. 

The  first  objective  is  achieved  by  applying  a discount  factor  to  the  investment 
and  cost/savings  streams  such  that  the  discounted  present  value  of  the  two 
streams  is  equal.  This  rate  of  return  on  investment  can  be  compared  with  the 
same  calculations  made  for  alternative  investments  to  aid  in  making  a selection 
decision.  If  the  calculated  value  for  any  single  proposal  is  below  some  accept- 
able threshold  value,  the  decision  to  reject  the  proposal  may  be  warranted.  A 
similar  approach  can  be  taken  by  selecting  the  threshold  discount  rate  and 
applying  it  to  the  investment  and  cost/savings  streams.  The  difference  or  Net 
Present  Value  (NPV)  of  these  two,  if  positive,  provides  another  measure  of  worth 
for  undertaking  the  project.  As  with  the  rate  of  return  on  investment  figure, 
the  NPV  can  be  compared  among  investment  alternatives  for  making  a selection 
decision.  Two  other  figures  useful  for  comparing  alternatives  are  (1)  the 
Savings  Investment  Ratio  which  will  be  some  figure  greater  than  one  if  the  NPV 
were  positive,  and  (2)  the  Uniform  Annual  Cost  that  provides  the  average  annual 
cost  which  is  useful  for  comparing  alternatives  which  have  differing  useful 
lives,  i.e.,  different  planning  periods. 

The  second  objective  is  achieved  by  taking  mutually  exclusive  alternatives  and 
calculating  the  previously  identified  figures  of  merit  for  differences  in  invest- 
ment and  cost/savings  streams  of  each  separate  alternative.  This  provides 
figures  of  merit  on  the  incremental  investment  required  of  one  alternative  over 
another. 

These  calculated  values,  plus  the  component  values  of  the  total  investment  and 
total  recurring  cost/savings,  are  presented  in  several  financial  report.. 

ETAM  Alternative  Analysis  Report  (FAQ! ) . The  following  identifies  data  sources 
and  calculational  logic  for  generating  the  fields  on  the  ETAM  Alternative 
Analysis  Report.  Fields  outlined  on  the  report  layout  not  described  here  are 
fixed,  and  are  generated  by  the  program  format  statements.  A sample  report 
showing  data  elements  by  column/row  coordinate  -is  shown  in  Figure  III-57. 
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FIGURE  111— Sa.  ETAM  FINANCIAL  ANALYSIS  (FIN)  PROCESS  FLOW 
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FIGURE  111-57.  FORMAT  OF  ETAM  ALTERNATIVE  ANALYSIS  REPORT 

(FA01) 


inwu  nLrimi  DU.  4U 


COL/ ROW  VARIABLE 


1C 

- DATE 

IE 

- PROJA 

2E  - PROJB 


IF  - YSRA 


2F  - YRSB 


1J  - INVNDA 


2J  - INVNDB 


3J  - INVNOT 

IK  - RECNDA 

2K  > RECNDB 

3K  - RECNDT 


OATA  SOURCE  AND/OR  CALCULATION  LOGIC 

This  field  Is  generated  within  the  system  and  developed 
as  MM/DD/YY. 

This  field  represents  the  Project  Number  of  the  primary 
project  under  analysis  and  to  which  alternative  projects 
will  be  compared.  It  will  be  passed  to  this  routine  as  a 
parameter. 

This  field  Identifies  an  alternative  project  being  com- 
pared. It  may  be  Identified  by  the  user  based  upon  a 
review  of  project  numbers  stored  within  the  ALTP  (Alter- 
native Project)  file  In  the  Project  Data  Base.  A para- 
meter Is  generated  as  In  IE  above. 

This  field  Identifies  the  number  of  years  over  which  the 
project  (PROJA)  has  been  analyzed.  Its  source  is  the 
CMR  (Cost  Model  Results)  project  file  for  PROJA.  Where 
the  CMR  contains  output  from  more  than  one  cost  model  and 
the  years  of  project  life  are  different,  the  greater 
number  of  years  should  be  used. 

This  field  Identifies  the  number  of  years  over  which  the 
project  (PROJB)  has  been  analyzed.  Its  source  Is  the 
CMR  project  file  for  PROJB.  Where  the  CMR  contains  out- 
put from  more  than  one  cost  model  and  the  years  of 
project  life  are  different,  the  greater  number  of  years 
should  be  used. 

This  field  contains  the  sum  of  non-dlscounted  annual 
Investments  for  PROJA  from  the  CMR  for  that  project. 

This  sum  may  contain  the  results  of  multiple  cost  model 
outputs  as  well  as  any  Investment  from  undertaking  risk 
reduction  projects. 

This  field  contains  the  sum  of  non-dlscounted  annual 
Investments  for  PROJB  from  the  CMR  for  that  project. 

This  surn  may  contain  the  results  of  multiple  cost  model 
outputs  as  well  as  any  Investment  from  undertaking  risk 
reduction  projects. 

INVNDA  - INVNDB 

This  field  contains  the  sum  of  non-dlscounted  annual 
recurring  costs  and  saving  for  PROJA  from  the  CMR  for 
that  project.  This  may  contain  the  results  of  multiple 
cost  model  outputs. 

This  field  contains  the  sum  of  non-dlscounted  annual 
recurring  costs  and  savings  for  PROJB  from  the  CMR  for 
that  project.'  This  may  contain  the  results  of  multiple 
cost  model  outputs. 

RECNDA-RECNDB 
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1M  - INVPVA 


2M  - INVPVB 


3M  - INVPVT 

IN  - ENDPVA 


2N  - ENDPVB 


3N  - ENDPVT 

IP  - INVTA 

2P  - INVTB 


This  field  contains  the  sum  of  discounted  annual  invest- 
ments for  PROJA.  This  is  the  present  value  (PV)  of  the 
stream  of  investments  summed  into  INVNDA  and  is  calcu- 
lated as: 

p+r 

(-  2(T+r)d)  for  n = 1 to  the  value  of  (YRSA  -1). 

The  value  of  r (discount  rate)  is  either  inputted  by  the 
assessor  or  defaults  to  .10  (10%). 

This  field  contains  the  sum  of  discounted  annual  invest- 
ments for  PROJB.  This  is  the  present  value  (PV)  of  the 
stream  of  investments  summed  irto  INVNDB , and  is  calcu- 
lated as: 

(-  2 (V+r)n^  ^0r  n = 1 t0  the  value  of  (YRSB  -1). 

The  value  of  r (discount  rate)  must  be  the  same  as  used 
for  INVPVA. 

INVPVA  - INVPVB  m 

| 

This  field  contains  the  end  (or  salvage)  value,  discounted, 
for  the  PROJA  investments  included  in  INVPVA.  Its  source 
is  the  final  year  (YRSA)  of  the  investment  stream  (INVNDA), 
discounted  as:  2+r  where  n = YRSB  and  r is  the  discount 

2(l+r)n 

rate  used  for  INVPVA. 

This  field  contains  the  end  (or  salvage)  value,  discounted, 
for  the  PROJB  investments  included  in  INVPVB.  Its  source 
is  the  final  year  (YRSB)  or  the  investment  stream  (INVNDB), 
discounted  as:  2+r  where  n = YRSB  and  r is  the  discount 

2(l+r)ri 

rate  used  for  INVPVA. 

ENDPVA  - ENDPVB 
INVPVA  - ENDPVA 
INVPVB  - ENDPVB 


3P  - INVTT 


INVTA  - INVTB 


1R 


2R 


3R 


RECPVA 


RECPVB 


RECPVT 


This  field  contains  the  sum  of  discounted  annual  recur- 
ring costs  and  savings  for  PROJA.  This  is  the  present 
value  (PV)  of  the  stream  of  recurring  costs/savings 
summed  into  RECNDA.  It  is  calculated  similar  to  INVPVA. 

This  field  contains  the  sum  of  discounted  annual  recur- 
ring costs  and  saving  for  PROJB.  This  is  the  present 
value  (PV)  of  the  stream  of  recurring  costs/savings 
summed  into  RECNDB . It  is  calculated  similar  to  INVPVB. 


RECPVA  - RECPVB 
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IT  - DISCA 

2T  - DISCB 

3T  - DISCT 

1U  - SIVRA 

2U  - SIVRA 

311  - SIVRT 

IV  - RORA 

2V  - RORB 

3V  - RORT 

1W  - UNIFA 
2W  - UNIFB 

3W  - UNIFT 


This  field  contains  the  discount  rate  applied  to  the 
various  cost/savings  streams  In  PROJA  to  calculate  each 
of  the  present  values.  Its  source  Is  the  assessor  input 
for  prior  calculations  or  the  default  value  of  .10  (10%). 

This  field  contains  the  discount  rate  applied  to  the 
various  cost/savings  streams  In  PROJB  to  calculate  °ach 
of  the  present  values.  It  Is  equal  to  DISCA. 

DISCA  * DISCB  (both  PROJA  and  PROJB  have  been  discounted 

at  same  rate). 

RECPVA/INVTA 
RECPVB/INVTB 
RECPVT/INVTT 

This  field  Is  calculated  by  finding  the  value  r in  the 

equation  2+r  (where  n - year)  which  applied  to  the 
2(l+r)n 

Investment  stream  summed  Into  INVNDA  and  to  the  recurring 
costs/savings  stream  summed  Into  RECNDA  makes  the  discounted 
values  equal  to  each  other.  The  CMR  project  file  for  PROJB 
contains  both  vectors  to  be  discounted. 

This  field  Is  calculated  by  finding  the  value  r In  the 

equation  2+r  (where  n * year)  which  applied  to  the 

2(l+r)n 

Investment  stream  summed  into  INVNDB  and  to  the  recurring 
costs/savings  stream  summed  Into  RECNDB  makes  the  discounted 
values  equal  to  each  other.  The  CMR  project  file  for 
PROJB  contains  both  vectors  to  be  discounted. 

This  field  Is  calculated  by  finding  the  value  r in  the 

equation  2+r  (where  n ■ year)  which  applied  to  the 

2(l+r)n 

vector  difference  (INVNDA*  - INVNDB*)  of  Investments  and 
to  the  vector  difference  (RECNDA*  - RECNDB*)  for  recurring 
cost/savings,  makes  the  discounted  values  equal  to  each 
other.  As  previously  outlined,  the  CMR  project  files  for 
PROJA  and  PROJB  contain  these  vectors. 

INVTA/  ((2+r)/2  (l+r)n)n  - i to  YRSA  where  YRSA  Is  the 
planning  period  for  PROJA  and  r Is  the  rate  at  w+ch  Invest- 
ment and  cost/savings  streams  were  discounted. 

INVTB/  ( (2+r)/2(l+r)n)n  ■ 1 to  YRSB  where  YRSB  is  tne 
planning  period  for  PROJB  and  r is  the  rate  at  which  invest- 
ment and  cost/savings  streams  ».jre  discounted. 

• 

INVTT/  ((2+r)/2(l+r)n)n  - 1 to  I where  I Is  the  greater 
of  the  planning  periods  for  either  PROJA  or  PROJB,  and  r 
Is  the  rate  at  which  Investment  and  cost/savings  streams 
were  discounted. 
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The  results  of  each  analysis  of  an  alternative  project  against  the  primary 
project  is  stored  in  the  FINR  (Financial  Results)  project  file.  The  entire  F I NR 
project  file  for  the  project  under  analysis  routine  is  called  to  calculate  new 
FA01  data,  if,  and  only  if,  a different  discount  rate  is  used.  Otherwise  the 
new  or  repeat  analysis  due  to  an  addition  or  change  of  an  alternative  project  is 
entered  into  the  FINR  project  file  as  a replacement  if  the  alternative  project 
number  already  exists,  or  as  an  addition  if  the  alternative  project  number  is 
new.  The  data  elements  in  Column  1 of  FA01  report  are  stored  only  once;  the 
data  elements  in  Column  3 are  stored  from  each  run  of  an  FA01  report.  The  FINR 
data  elements  are  essentially  those  which  appear  on  the  ETAM  Alternative  Analysis 
Summary  Report  (FA02).  These  data  elements  stored  within  the  FINR  project  file 
are: 


1C  - 
IE  - 

DATE 

PROJA 

2E 

IF  - 

YRSA 

2F 

IP  - 

INVTA 

3P 

1R  - 

RECPVA 

3R 

IT  - 
IV  - 

DISCA 

RORA 

3 V 

1W  - 

UN  I FA 

3W 

- PROJB  (1-5  if  available) 

- YRSB  (1-5  if  available) 

- INVTT  (1-5  if  available) 

- RECPVT  (1-5  if  available) 

- RORT  (1-5  if  available) 

- UNIFT  (1-5  if  available) 


1 1 1-143 


TAEG  REPORT  NO.  40 


FAQ!  Variable  List  Summary. 


Var  Name 

1L o 

Source/Destination 

Report  Ref 

DATE 

0 

INT  GEN/FINR+RPT 

1C 

PROJA-B 

I 

EXEC  PARAM/FINR+RPT 

1E-2E 

YRSA-B 

I 

CMR/FINR+RPT 

1F-2F 

INVNDA-B 

1 

CMR/RPT 

1J-2J 

INVNDT 

0 

CALC/RPT 

3J 

RECNDA-B 

I 

CMR/RPT 

1K-2K 

RECNDT 

0 

CMR/RPT 

3K 

INVPVA-B-T 

0 

CALC/RPT 

1M-3M 

ENDPVA-B-T 

0 

CALC/RPT 

1N-3N 

INVTA-T 

0 

■ CALC/F1NR+RPT 

1P+3P 

INVTB 

0 

CALC/RPT 

2P 

RECPVA-T 

0 

CALC/FiNR+RPT 

1R+3R 

RECPVB 

0 

CALC/RPT 

2R 

DISCA 

I 

INTERACTIVE/FINR+RPT 

IT 

DISCB-T 

0 

CALC/RPT 

2T-3T 

SIVRA-B-T 

0 

CALC/RPT 

1U-3U 

RORA-T 

0 

CALC/FINR+RPT 

1V+3V 

RORB 

0 

CALC/RPT 

2 V 

UNIFA-T 

0 

CALC/FINR+RPT 

1W+3W 

UNIFB 

0 

CALC/RPT 

2W 
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ETAM  Alternative  Analysis  Summary  Report  (FA02).  The  following  identifies  the 
data  sources  and  calculational  logic  for  generating  the  fields  on  the  ETAM 
Alternative  Analysis  Summary  Report.  Fields  outlined  on  the  report  layout  not 
described  here  are  fixed,  and  are  generated  by  the  program  format  statements. 

A sample  report  showing  data  elements  by  column/row  coordinate  is  shown  in 
Figure  1 1 1-58 . 

COL/ ROW  VAR  DATA  SOURCE  AND/OR  CALCULATIONAL  LOGIC 

3B  - DATE  This  field  is  operated  within  the  svstem  and  developed 

as  MM/DD/YY. 

1G  - PROJA  This  field  represents  the  Project  Number  of  the  primary 

project  under  analysis  and  to  which  Alternative  Projects 
will  be  compared.  It  is  stored  within  the  FINR  project 
file  name  (fn).  (Same  as  FA01  1 E-PROJA) 

1H  - YRSA  This  field  identifies  the  number  of  years  over  which  the 

project  (PROJA)  has  been  analyzed.  It  is  stored  within  the 
FINR  project  file.  (Same  as  FA01  1F-YRSA) 

IK  - INVTAK  This  field  contains  the  Total  Present  Value  of  the  New 

Investment.  It  is  calculated  by  taking  the  INVTA  (Same 
as  FA01  1P-INVTA)  in  the  FINR  project  file  and  dividing 
by  1000.  ( INVITA/100C) . 

1L  - RECPVAK  This  field  contains  the  Total  Present  Value  of  the 

Recurring  Cost/Savings  stream.  It  is  calculated  by  taking 
the  RECPVA  (Same  as  FA01  1R-RECPVA)  in  the  FINR  project 
file  and  dividing  by  1000  (RECPVA/1000) . 

1M  - NPVAK  = RECPVAK- INVPVAK 

IP  - DISCA  This  field  contains  the  discount  rate  used  to  calculate 

the  FA01  report  and  is  obtained  from  the  FINR  project 
file.  (Same  as  FA01  1T-DISCA). 

IQ  - SIRVA  = RECPVAK/ INVTAK 

1R  - RORA  This  field  contains  the  Rate  of  Return  on  Investment 

calculated  for  PROJA  and  stored  in  the  FINR  project  file 
(same  as  FA01  1V-R0RA). 

IS  - UN  I FA  This  field  contains  the  Uniform  Annual  Investment  Cost 

calculated  for  PROJA  and  stored  in  the  FINR  project  file 
(same  as  FA01  1W-UNIFA). 

2G  - PR0JB1  This  field  identifies  an  alternative  project  which 

3G  - PR0JB2  was  compared  to  the  primary  project  and  outputted  on 

4G  - PR0JB3  an  FA01  report.  It  will  have  appeared  as  a PROJB  on 

5G  - PR0JB4  that  report.  It  is  stored  within  the  FINR  project 

6G  - PR0JB5  file. 
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2H 

- 

YRSB1 

This  field  identifies  the  number  of  years  the  project 

3H 

- 

YRSB2 

was  analyzed.  It  will  have  appeared  as  a YRSB  on  the 

4H 

- 

YRSB3 

FA01  report.  It  is  stored  within  the  FINR  project  file. 

5H 

- 

YRSB4 

6H 

- 

YRSB5 

2K 

INVTT1 

This  field  contains  the  Net  Difference  in  the  Total 

3K 

- 

INVTT2 

Present  Value  of  the  New  Investment  for  the  Primary 

4K 

- 

INVTT3 

Project  and  the  particular  Alternative  Project.  It 

5K 

- 

INVTT4 

is  calculated  by  taking  the  INVTT  (same  as  FA01  3P-INVTT) 

6K 

- 

INVTT5 

in  the  FINR  project  file  and  dividing  by  1000  ( I NVTT/1 000 ) . 

2J 

_ 

RECPVT1 

This  field  contains  the  Net  Difference  in  the  Total  Present 

3J 

- 

RECPVT2 

Value  of  the  Recurring  Cost/Savings  for  the  Primary  Project 

4J 

- 

RECPVT3 

and  the  particular  Alternative  Project.  It  is  calculated 

50 

- 

RECPVT4 

by  taking  the  RECPVT  (same  as  FA01  3R-RECPVT)  in  the  FINR 

6J 

- 

RECPVT5 

project  file  and  dividing  by  1000  (RECPCT/1000) . 

2M 

NPVT1 

= 

RECPVT1 -INVTT1 

3M 

- 

NPVT2 

= 

RECPVT2-INVTT2 

4M 

- 

NPVT3 

= 

RECPVT3-INVTT3 

5M 

- 

NPVT4 

= 

RECPVT4-INVTT4 

6M 

- 

NPVT5 

= 

RECPVT5-INVTT5 

2P 

_ 

DISCT1 

S 

DISCA 

3P 

- 

DISCT2 

= 

DISCA 

4P 

- 

DISCT3 

= 

OISCA 

5P 

- 

DISCT4 

= 

DISCA 

6P 

- 

DISCT5 

= 

DISCA 

2Q 

- 

SIVRT1 

= 

RECPVT1 / INVTT1 

3Q 

- 

SIVRT2 

= 

RECPVT2/INVTT2 

4Q 

- 

SIVRT3 

= 

RECPVT3/INVTT3 

5Q 

- 

SIVRT4 

= 

RECPVT4/INVTT4 

6Q 

- 

SIVRT5 

= 

RECPVT5/INVTT5 

2R 

_ 

R0RT1 

This  field  contains  the  Rate  of  Return  on  Investment 

3R 

- 

R0RT2 

calculated  for  the  Net  Difference  in  the  Investment  and 

4R 

- 

R0RT3 

Recurring  Cost/Savings  vectors  for  the  primary  and  alter- 

5R 

- 

R0RT4 

native  project,  and  stored  in  the  FINR  project  file 

6R 

- 

R0RT5 

(same  as  FA01  3V-R0RT) . 

2S 

- 

UNIFT1 

This  field  contains  the  Uniform  Annual  Investment.  Cost 

3S 

- 

UNIFT2 

calculated  for  the  Net  Difference  between  the  Primary 

4S 

- 

UNIFT3 

Project  and  an  Alternative  Project,  and  stored  in  the 

5S 

- 

UNIPT4 

FINR  project  file  (same  as  FA01  3W-UNIFT) . 

6S 

- 

UNIFT5 
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FA02  Variable  Summary. 


Var  Name 

I/O 

Source/Destination 

Report  Ref 

DATE 

0 

INT  GEN/RPT 

3B 

PROJA 

I 

FINR/RPT 

1G 

YRSA 

I 

FINR/RPT 

1H 

INVTAK 

0 

CALC/RPT 

IK 

RECPVAK 

0 

CALC/RPT 

1L 

NPVAK 

0 

CALC/RPT 

1M 

DISCA 

I 

FINR/RPT 

IP 

SIRVA 

0 

CALC/RPT 

IQ 

RORA 

I 

FINR/RPT 

1R 

UNIFA 

I 

FINR/RPT 

IS 

PR0JB1-5 

I 

FINR/RPT 

2G-6G 

YRSB1-5 

I 

FINR/RPT 

2H-6H 

INVTT1 -5 

0 

CALC/RPT 

2K-6K 

RECPVT1 -5 

0 

CALC/RPT 

2J-60 

NPVT1-5 

0 

CALC/RPT 

2M-6M 

DISCT1-5 

0 

CALC/RPT 

2P-6P 

SIVRT1 -5 

0 

CALC/RPT 

2Q-6Q 

R0RT1-5 

I 

FINR/RPT 

2R-6R 

UNIFT1-5 

I 

FINR/RPT 

2S-6S 

i 


3 


! 

1 
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Rate  of  Return  on  Investment  Algorithms.  The  calculation  of  the  Rate  of  Return 
on  Investment  required  in  the  Financial  Analysis  routine  is  performed  by  an 
iterative  algorithm  as  follows: 

1.  Make  an  initial  estimate  of  the  Rate  of  Return  (RORE)  by  assuming 
the  total  investment  takes  place  at  time  t = 4 and  that  the  savings 
stream  (RECND*)  occurs  in  equal  installments  over  the  planning 
period  (N) 

The  cumulative  annual  discount  factor  (RCUM)  would  discount  an  equal 
amount  savings  stream  (RECND*)/N)  to  the  value  of  the  investment  stream 
(INVND*)  is  given  by 

a)  RCUM  = (INVND*  * N)/RECND* 


A polynomial  of  order  N must  be  solved  for  the  Rate  of  Return  to 
establish  the  initial  value.  The  polynomial  is  defined  as: 


N)  + 

if  N! 
VV(N-k) !k! 


N!  \R0RE 

[N-k-1 ) ! (k+1 ) l) 


RCUM  * R0REN  =0 

If  the  savings  stream  were  actually  in  equal  annual  amounts,  and  the 
investment  all  took  place  at  t = 0,  this  calculation  will  produce  an 
RORE  which  will  be  higher  than  the  true  rate  since  the  polynomial  is 
based  upon  the  formula  for  the  savings  occuring  at  the  end  of  each 
year;  i.e.: 


c)  k=N 

y~  l/(HR0RE)k, 


rather  than  the  formula  for  the  savings  occurring  throughout  the 
year,  i.e.: 

d)  k=N 

(2+R0RE)/(2(l+R0RE;  ) 

An  algorithm  for  evaluating  the  polynomial  b)  is  shown  in  the  following 
flow,  Figure  1 1 1 - 59 . 

2.  Since  the  initial  estimate  will  generally  be  slightly  high,  choose  a 

value  approximately  .01  (1%)  lower  than  the  RORE  solution  from  1.  above. 
Using  the  formula  l.d),  discount  both  the  investment  stream  (INVND*)  and 
the  recurring  savings  stream  (RECND*)  over  the  planning  period  (N). 


i 
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a) 


INVPV 


RECPV 


* (2+R0RE)/(2(l+R0RE)k) 

* (2+R0RE)/(2(l+R0RE)k) 


If  RECPV  Is  greater  than  the  Investment  INVPV  then  the  value  of 
RORE  Is  too  small  and  should  be  Increased  by  .005  (.5*).  If  RECPV 
Is  less,  then  reduce  RORE  by  .01  (1*).  Repeat  the  process  by  halving 
the  Increment/decrement  values  on  each  successive  iteration  until  RORE 
Is  calculated  to  the  desired  level  of  accuracy.  To  ensure  that  a con- 
tinuous loop  Is  not  encountered,  a suitable  test  should  be  set  up  to 
cause  an  exit  from  the  routine.  An  example  of  an  Iterative  sequence 
Is  shown  In  the  following  flow.  Figure  I I 1-59 . 


Program  P62  (Financial  Analysis  Setup). 

Purpose:  This  program  establishes  the  run  parameters  for  the  Financial  Analysis 

calculations  and  reports. to  be  performed  by  program  P63.  It  provides 
the  user  with  Interactive  capability  to  produce  several  of  the  Project 
Data  Base  files,  to  run  the  Financial  Analysis  Program,  or  to  exit 
routine.  The  analysis  requirements  are  established  by  Identifying  the 
alternate  projects  which  are  to  be  financially  analyzed.  The  option  to 
analyze  all  alternate  projects  In  relation  to  the  primary  project  Is 
available  to  the  user. 


Inputs:  The  primary  Inputs  to  this  program  are: 

o ALTP  file  for  Identification  of  alternative  projects, 

o User  Interactive  terminal  Input  to  control  the  processing  sequence. 

Process:  The  following  are  the  major  process  steps  for  Program  P62: 

1.  The  ALTP  file  Is  analyzed  to  determine  If  alternate  project  ID's 
exist  and.  If  they  represent  valid  project  Identifiers.  Any 
Invalid  project  Identifiers  are  displayed. 

2.  The  user  receives  a display  of  the  select  options.  They  are  as 

o Display  ALTP  file 

o Review  ID  file  of  selected  ALTP  project  ID 

o Run  Financial  Analysis 

o Quit 

3.  If  the  user  selects  either  of  the  dislay  or  review  options, 

Program  PI  Is  called  In  order  to  display  the  contents  of  either 
the  ALTP  file  or  the  contents  of  the  ID  file  represented  by  the 
alternative  project  Identifiers. 

4.  If  the  Run  Financial  Analysis  option  Is  selected,  the  user  Is 
displayed  three  options,  namely: 

o Single  analysis 

o Analyze  all 

o Quit 
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5.  The  user  selects  one  of  these  options  and  the  program  sets  up  the 
run  conditions  for  entering  the  financial  analysis  Program  P63. 
Selection  of  the  single  analysis  option  will  cause  only  an  FA01 
Financial  Analysis  Report  comparing  the  primary  project  with  the 
selected  ALTP  project  to  be  printed.  Selection  of  the  Analyze 
All  option  will  cause  Program  P63  to  print  an  FA01  report  for 
each  ALTP  within  the  ALTP  file  and  a FA02  Financial  Analysis 
Summary  report. 

Outputs:  The  major  output  of  Program  P62  is  a series  of  displays  which  request 

the  user  to  enter  an  option  for  controlling  the  processing  sequence 
of  the  reports  generated  as  a result  of  the  selected  sequences  are 
generated  by  either  Program  Pi  or  Program  P63. 

Program  P63  (Financial  Analysis  Calculations). 

Purpose:  The  purpose  of  this  program  is  to  perform  the  financial  calculations 

necessary  to  determine  several  financial  measurement  factors,  and 
to  print  comparative  and  summary  reports  on  the  projects  selected  for 
comparison  with  the  primary  project. 

Inputs:  The  primary  Inputs  to  this  program  are: 

1.  The  run  parameters  established  by  Program  P62. 

2.  The  cost  model  results  data  previously  established  by  the  cost 
model  runs  and  the  Decision  Analysis  Program  P60. 

Process:  The  process  steps  required  to  perform  the  financial  calculations  were 

previously  outlined  In  the  sections  entitled  ETAM  Alternative  Analysis 
Report  (FA01)  and  ETAM  Alternative  Analysis  Summary  Report  (FA02). 

A computational  algorithm  for  computing  Rate  of  Return  on  Investment 
is  presented  In  a separate  section  following  the  outTIne  of  the  FA02 
report. 

Outputs:  The  outputs  of  Program  P63  are: 

1.  An  ETAM  Financial  Analysis  Report  ( F AO 1 ) 

2.  An  ETAM  Financial  Analysis  Summary  Report  (FA02). 

3.  The  results  of  each  of  the  financial  comparisons  calculated  by 
this  program  are  stored  In  the  FINR  file. 
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SECTION  IV 


CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS 

The  following  conclusions  are  based  upon  a rather  broad  investigation  into  the 
use  of  models  and  procedural  techniques  in  the  decision  making  process.  Much  of 
the  literature  deals  with  highly  theoretic  concepts  which  are  difficult  to  apply 
to  the  pragmatic  requirements  of  making  and  implementing  a decision.  These  con- 
clusions are  based  upon  the  level  of  design  and  development  achieved  to  this 
point  on  the  Educational  Technology  Assessment  Model.  Range-of-Effect  was 
developed  to  a functioning  level,  however,  the  Abbreviated  Data  Bases  which  are 
the  objects  of  the  Range-of-Effect  search  contain  a limited  subset  of  the 
Courses,  Vehicles,  and  Job/Tasks  relative  to  naval  training  and  operations. 

Thus,  these  conclusions  do  not  reflect  any  substantial  use  of  a computerized 
version  of  ETAM,  but  rather  consider  the  techniques  inherent  in  the  application 
of  ETAM  to  the  general  decision-making  process. 

ETAM  continues  to  appear  a robust  model  of  rational  decision  making  in  general 
and  for  changes  (innovations)  in  training  technology  in  particular.  Its 
strengths  come  from  two  major  factors  that  have  dominated  its  development: 

(a)  the  basic  tasks  in  the  analytic  and  decision-making  sequence  are  proto- 
typical of  informed  conmon  sense  rationality,  and  (b)  the  computer  is  used  not 
to  hide  or  obscure  this  rational  structure,  but  to  help  the  humans  to  apply  it 
by  exposing  it.  The  computer  is  a convenience,  at  least  in  concept.  In  prac- 
tice, because  of  the  massive  amounts  of  word  data  and  numerical  data  to  be 
manipulated,  if  the  procedures  are  to  be  applied  in  their  full  scale,  this  con- 
venience factor  becomes  magnified  into  a virtual  necessity. 

The  taxonomic  structures  for  describing  Courses,  Vehicles,  and  Job/Tasks  provide 
important  discriminants  for  searching  out  potential  applications  for  the 
innovation.  The  coding  of  the  sample  entities  showed  some  definite  clustering 
of  descriptors  which  might  be  expected  due  to  the  similarities  in  Navy  training 
courses;e.g. , maintenance,  procedural,  operator,  etc.  Refinements  in  the 
techniques  for  using  the  descriptor  set  should  improve  search  validity. 

Since  the  Range-of-Effect  program  development  was  completed,  it  was  possible 
to  determine  the  operational  costs,  at  least  in  the  application  of  the  pro- 
grams to  the  limited  subsets  of  entitles  which  were  described  in  the  Abbrevi- 
ated Data  Bases.  Following  are  costs  of  a typical  Range-of-Effect  sequence: 


Load  Project  Data  Base  (Program  P17)  $ 7 - $ 8 

Edit  Search  Descriptors  (Program  P2)  $ 4 - $ 5 

Search  Abbreviated  Data  Bases  (Program  P3)  3-4 

Print  Search  Results  (Program  P5A)  3 - 4 

Edit  Search  Results  (Program  P5B)  3 - 4 

Print  Extract  Search  Results  (Program  P5C)  _2  - _3 

Total  Range-of-Effect  Processing  Cost  $15  - $20 
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Because  the  Assessment  function  within  ETAM  was  completed  only  to  a design 
level,  It  Is  not  possible  to  predict  typical  operational  costs  for  the  pro- 
cessing sequences  making  up  that  function. 

Extensive  reports  have  been  previously  published  describing  the  ETAM  procedures. 
However,  It  was  believed  that  It  would  be  useful  to  the  reader  to  reemphasize 
the  basic  ETAM  functions  by  contrasting  the  procedures  as  they  were  originally 
envisioned  for  ETAM  and  supported  by  computer  processing,  with  the  "normal" 
reasonable  way  It  would  be  performed.  Following  are  summaries  of  the  two  ap- 
proaches In  contrast  with  one  another. 

TASK  1 - FORMALIZE  THE  DESCRIPTION  OF  THE  INNOVATION. 

Normal  Procedure  An  Informal  prose  statement  Is  made  about  what  the  Innovation 
or  change  Is  about,  what  It  consists  of,  and  descriptions  of  local  effects  In 
limited  application  environments.  An  Interoffice  project  file  may  or  may  not  be 
created. 

ETAM  Procedure.  A structured  format  reminds  the  proposer  of  the  Innovation  to 
set  down  all  categories  of  Information  relevant  to  the  Innovation  that  dredges 
what  the  Innovator  has  learned  or  knows  about  the  costs,  relevance,  benefits, 
practical  factors,  supporting  data,  and  expected  R&D  effort.  A formal  project 
file  Is  Initiated.  The  category  format  will  enable  direct  referencing  at  later 
stages  of  Investigation  to  these  starting  data. 

TASK  2 - DEVELOP/EXAMINE  ALTERNATIVES  TO  THE  INNOVATION. 

Normal  Procedure.  The  proposal  may  have  focused  or  defined  an  organiza- 
tional need  or  problem.  Discussions  may  develop  competing  proposals. 

Adversary  relationships  may  produce  arguments  that  blur  the  overall  pic- 
ture. The  alternatives  may  or  may  not  be  formally  recorded.  They  prob- 
ably do  not  get  associated  with  the  project  file. 

ETAM  Procedure.  Alternatives  are  sought  formally  and  documented  according 
to  the  same  format  used  for  the  source  proposal.  A deliberate  attempt 
Is  made  to  achieve  a simpler,  less  costly  or  less  risky  solution  that 
may  be  less  disruptive  to  existing  or  projected  operations.  Promising 
alternatives  can  be  put  through  later  ETAM  procedures  and  compared 
directly,  variable  for  variable,  with  each  other  and  the  source  proposal 
at  decision  stages. 

TASK  3 - MAKE  PRELIMINARY  FEASIBILITY  PROFILE. 

Normal  Procedure.  This  step  may  be  performed  Informally  In  verbal  dls- 
cusslons  but  without  clear  and  documented  outcomes.  One  or  two  Issues 
may  lead  to  neglect  of  the  others.  The  factors  Include  Importance  of 
the  outcome  to  the  organizational  mission,  compatablllty  with  organiza- 
tional policy,  operations  and  goals,  state-of-the-art,  R&D  funding, 
technical  support,  and  attltudlnal  acceptance.  The  overall  Impression 
may  be  so  negative  that  the  proposal  Is  dropped. 


ETAM  Procedure.  The  major  feasibility  factors  are  all  covered  and  experts 
express  their  opinions  on  attitude  scales  which  become  essential  documented 
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references  that  formally  enter  into  the  decision  structure.  The  summary 
of  feasibility  may  indicate  clearly  that  the  proposal  should  now  be  dropped 
without  further  analysis.  The  clearly  expressed  evaluations  on  each  sig- 
nificant variable  may  be  revised  in  the  light  of  further  information  that 
has  direct  bearing.  All  data,  including  revisions,  are  stored  in  the 
project  file. 

TASK  4 - PERFORM  ANALYTIC  FEASIBILITY  ASSESSMENT. 

Normal  Procedure.  Attention  focuses  on  the  weak  spots  and  potentially 
high-risk  areas  in  the  proposal,  especially  as  they  involve  costs  and 
size  of  risk,  activities  that  might  ameliorate  the  risks  but  having  a 
cost  reference.  Informed  guesses  may  be  made  as  to  the  range  of  effective 
usefulness  of  the  innovation  in  order  better  to  prorate  the  costs  of  im- 
plementation and  acceptance.  Tentative  feasibility  may  appear  or  disappear 
at  this  stage.  The  project  may  be  dissipated  in  a sea  of  controversy. 

ETAM  Procedure.  Formal  procedures  are  used  for  assessing  grossly  esti- 
mated risks  and  costs  with  and  without  supportive  projects.  Cost  data 
are  entered  into  defined  categories  that  permit  direct  comparisons. 

Decision  trees  are  developed  from  utility  scaling  procedures  in  order  to 
obtain  figures  of  merit  for  various  outcomes.  The  operations  for  gener- 
ating the  source  data  and  their  conversion  by  human  judgment  into  estimates 
of  worth  are  explicit,  and  enter  the  project  file. 

TASK  5 - DETERMINE  RANGE-OF-EFFECT. 

Normal  Procedure.  If  the  proposal  overcomes  preliminary  hurdles,  further 
exploration  is  made  into  the  range  of  situations  or  entities  that  could 
be  affected  beneficially  or  otherwise  by  the  innovation.  Because  of  the 
difficulty  of  expressing  intangible  benefits  or  liabilities  in  the  opera- 
tional context,  they  tend  either  to  be  lost  or  over-inflated.  The  readi- 
ness with  which  directly  tangible  cost  savings  can  be  defended  tends  to 
make  for  organizational  conservatism  in  the  acceptance  of  innovations. 
Range-of-effect  determination  tends  to  be  limited  to  the  knowledge  of 
the  individual  staff  expert  about  the  operations  of  the  enterprise  and  to 
his  perception  of  the  functions  of  the  innovation. 

ETAM  Procedure.  A formal  structure  of  all  major  variables  that  go  into 
the  organizational  mission  is  available  to  the  analyst,  expressed  as  a 
taxonomy  for  applying  to  processes  (teaching  and  learning)  and  to  entities 
(courses,  devices,  job-tasks).  Entities  are  indexed  according  to  their 
key  properties  relevant  to  the  mission.  The  same  taxonomy  is  used  for 
indexing  with  descriptors  the  Innovation.  ETAM  has  built-in  instruction 
to  inform  and  guide  the  assessor  who  Is  less  than  omniscient.  Automatic 
search  and  retrieval  operations  assist  the  human  expert  in  determining 
the  full  range  of  entities  and  attributes  that  will  be  affected  by  the 
innovation.  The  listing  of  these  entities  can  enter  into  automatic  or 
manual  cost  analysis.  The  human  expert,  however,  makes  the  final  deci- 
sion of  relevance  about  each  entity  from  context  information  not  limited 
merely  to  the  matching  of  descriptors. 
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TASK  6 PERFORM  COST-BENEFIT  ANALYSIS. 

Normal  Procedure.  The  enterprise  may  have  formal  cost  models  for  Its  operations. 
These  are  now  brought  to  bear  on  each  of  the  relevant  entities  for  which  some 
proportional  cost  benefit  (negative  or  positive)  has  been  estimated.  The  wise 
estimator  puts  In  a safety  factor  for  his  estimates  by  expressing  a range  of 
likely  error  with  optimistic  and  pessimistic  limits.  Cost  data  are  summarized. 
Other  benefit  and  liability  Information  Is  presented  In  prose  discussions.  The 
paper  Is  apt  to  be  voluminous. 

ETAM  Procedure.  Entitles  Identified  by  range-of-effect  procedures  automatically 
enter  into  their  appropriate  cost  models,  and  costs  are  automatically  aggregated. 
Input  changes  are  therefore  readily  reflected  in  output  changes.  Utility  scaling 
enables  intangibles  to  be  assessed  on  the  same  continuum  of  worth  as  tangible 
(i.e.,  cost)  variables.  All  constituent  operations  are  overt  and  documented, 
and  can  be  related  back  to  their  source  data.  The  resulting  decision  tree  not 
only  displays  the  estimated  values  of  the  decision  alternatives,  it  also  serves 
as  a high  level  Index  for  interrogating  the  sources  of  the  conclusions.  Auto- 
mation of  the  calculational  model  enables  quantitative  evaluation  of  variables 
to  which  the  decision  outcomes  are  most  sensitive,  thus  enabling  focus  on  where 
major  risks  may  lie. 

TASK  7 PERFORM  FINANCIAL  ANALYSIS. 

Normal  Procedure.  The  investment  required  by  the  innovation  Is  now  viewed  in 
the  long-term  and  contrasted  with  alternative  opportunities  and  needs  for  invest- 
ments. This  concern  recognizes  that  any  organization  has  limited  capital  and 
competing  claims  on  that  capital.  Cost/savings  analysis  Is  made  for  each  alter- 
native, as  well  as  for  alternative  proposals.  The  objective  Is  to  reconcile 
organizational  priorities,  limited  capital,  risk,  and  proposals  for  Innovation 
with  their  respective  costs  and  promises  of  benefits. 

ETAM  Procedure.  This  is  a formal  step  with  the  same  objectives.  Multiyear 
projections  of  costs  and  savings  are  generated  by  machine  calculations.  The 
input  data  for  making  the  calculations  will  already  exist  as  content  in  the 
project  file  that  has  been  developed  In  performing  the  previous  tasks. 

TASK  8 MAKE  THE  ACCEPT/REJECT/STUDY  FURTHER  DECISION. 

Normal  Procedure.  The  executive  Is  presented  with  a brief  statement  of  the 
problem  and  innovation  and  bottom-line  summary  data.  These  may  be  preceded  by  a 
flip  chart  presentation  of  topic  statements  and  selected  summary  data.  The 
backup  may  be  a paper  file  of  hundreds  of  pages.  The  executive  sizes  up  the 
situation  intuitively,  asks  his  staff  some  "probing"  test  questions  and  arrives 
at  an  assessment  of  the  competence  with  which  the  staff  work  was  done.  He  makes 
an  intuitive  judgment  of  the  comparative  value  of  the  Innovation  In  the  organiza- 
tional scheme  of  things  and  makes  or  postpones  a decision.  On  occasion,  he  may 
spend  a weekend  trying  to  digest  the  backup  report. 

ETAM  Procedure.  The  executive  is  presented  with  a brief  statement  of  the  problem 
and  the  Innovation,  taken  from  Task  1.  If  he  wishes  more  detail  about  the 
source  data  on  the  innovation,  he  can  Invoke  the  file  on  Task  1.  He  may  request 
the  display  of  the  outcome.  The  decision  tree  and  financial  analysis  are  presented. 
He  requests  outputs  from  the  sensitivity  analyses  to  find  out  his  exposures  to 
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the  precision  of  the  estimates  and  judgments  of  the  assessing  staff.  These  are 
displayed  at  once.  He  may  prefer  to  use  his  own  intuitive  and  global  judgment, 
so  he  invokes  summaries  of  the  data  used  by  the  assessor  in  Task  6 in  developing 
the  decision  tree.  Or,  the  executive  challenges  the  implicit  set  of  organiza- 
tional values  held  by  the  assessing  staff  and,  invoking  the  utility  scaling 
operations  of  Task  6,  modifies  the  respective  utilities  of  several  variables  and 
immediately  reviews  the  outcome  on  the  recomputed  decision  tree.  He  may  question 
the  evaluations  about  feasibility  by  examining  the  structured  data  on  each  of 
the  topic  responses  in  Task  3.  In  the  space  of  an  hour,  the  executive  can  rove 
selectively  through  the  project  and  "peel  back"  from  summaries  to  data  content 
to  data  source. 

ETAM  does  not  take  away  any  of  the  traditional  prerogatives  of  the  executive, 
but  augments  them  and  enables  a telescoping  of  time  for  investigative  inquiry, 
rapid  selective  access,  and  standard  formatting  of  key  information. 

Delivering  this  capability  depends  upon  a suitable  display  terminal  and  an 
automatic  access  structure  that  is  not  a part  of  the  completed  ETAM  work.  But 
the  data  base  structures  now  being  devised  will  make  this  capability  possible. 

PROCEDURAL  SUMMARY.  The  parallels  between  normative  processes  for  arriving  at 
decisions  and  ETAM  procedures  should  be  evident.  It  may  be  suggested,  without 
emphasis,  that  political  pressures  may  as  readily  be  put  into  formal  utility 
models  as  any  other  variables  that  affect  desirability  of  one  outcome  over 
another. 

ETAM  represents,  however,  a shift  from  informal  or  semi-formal  methods  of  analysis 
and  synthesis  of  data  to  a formalism  of  what  data  are  acquired,  stored,  accessed, 
combined  with  other  data.  But  this  formalism  does  not  preclude  non-fcrmal 
processing  by  human  staff  and  executive  personnel.  ETAM  represents  orderliness 
and  organization  that  permits  efficiency  and  consistency  but,  more  important,  it 
enables  much  larger  amounts  of  pertinent  information  to  be  brought  to  bear  on 
the  decision  process.  In  effect,  a broader  scope  for  optimizing  (or  satisfying) 
is  brought  to  bear  with  a clearer  picture  of  the  range  of  consequences.  These 
factors  are  at  least  as  important  as  increases  in  the  efficiency  whereby  decisions 
are  made.  At  least  in  principle,  better  decisions  can  be  made. 

These  considerations  lead  to  the  conclusion  that  if  ETAM  were  not  developed  and 
used,  something  much  like  it  would  have  to  come  about  within  the  next  decade. 
Unfortunately,  computer  specialists  and  enthusiasts  generally  attempt  to  devise 
a universal  and  self-sufficient  automatic  formula  which  excludes  convenient 
human  intervention  and  participation  with  the  process.  When  this  approach  is 
shown  to  fail,  as  it  Inevitably  does,  the  resulting  cynicism  delays  sponsorship 
for  a more  sensible  iteration  on  the  problem. 

Even  without  further  work  in  the  indexing  and  linking  of  Navy  data  bases  or  in 
the  computerization  of  procedures,  ETAM  should  serve  as  a useful  tool.  Each  of 
the  eight  tasks  in  the  procedure  can  be  performed  manually.  Informed  estimates 
may  have  to  take  the  place  of  greatly  dispersed  sources  of  hard  data.  Specific 
programs  may  be  written,  or  adapted,  for  the  calculational  operations.  The 
files  may  consist  of  paper  documents  rather  than  electronic  store.  These  will 
dilute  the  efficiency  intended  for  ETAM  but  sustain  its  central  principles  and 
objectives  to  bring  the  largest  amount  of  relevant  data  feasible  to  the  decision 
process,  and  to  make  data  and  process  explicit. 
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The  range-of-effect  section  may  be  treated  as  a formal  and  comprehensive  struc- 
turing of  educational  and  training  technology,  and  therefore  a contribution  in 
its  own  right.  The  whole  of  training,  and  the  training  enterprise,  may  be  seen 
as  an  orderly  pattern  of  constituent  parts  and  objectives.  Needs  and  solutions 
may  be  perceived  In  large  contexts. 

In  the  sense  of  the  ETAM  task  sequence,  data  structures,  and  external  file 
disciplines,  ETAM  has  been  "completely"  described.  In  terms  of  Implementation 
in  terms  of  computerized  aids  and  efficiencies,  it  has  achieved  limited  realiza- 
tion with  limited  means.  In  terms  of  cost-effective  testing  under  whatever 
realistic  conditions  turn  out  to  be  feasible,  it  stands  only  on  the  threshold. 

REMAINING  PROBLEMS.  The  following  are  some  key  problems  that  affect  the  implemen- 
tation and  acceptance  of  ETAM.  (The  reader  may  notice  that  some  of  these  problems 
reflect  topics  in  ETAM  Task  3.) 

Indexing  the  Data  Bases.  Only  a sample  of  training  courses,  job-tasks,  and 
instructional  devices  have  been  indexed,  and  only  one  person  performed  the  task. 

At  a conservative  estimate,  two  persons  with  equivalent  skills  could  index  the 
entire  present  inventory  of  job-tasks  in  the  Occupational  Standards,  the  courses 
in  CANTRAC,  and  the  instructional  devices.  The  application  of  the  descriptors 
to  the  Range-of-Effect  task  remains  to  be  performed  so  that  the  reliability  of  the 
entities  obtained  during  a search  Gan  be  determined.  This  empirical  process 
should  be  performed  following  a more  extensive  incorporation  of  indexed  entities 
into  the  ETAM  Abbreviated  Data  Bases. 

Fine-Tuning  the  Procedures.  Only  rather  rudimentary  work  on  the  procedural 
interface  Between  computer  functions  and  human  users  will  have  been  completed  by 
the  termination  of  this  phase  of  the  project;  and  the  typewriter-like  terminal, 
as  contrasted  with  a graphic  CRT  display  terminal,  is  merely  an  expedient  stop- 
gap device  for  demonstrating  feasibility  rather  than  interactive  effectiveness. 

The  training  and  tutoring  of  the  assessor  at  the  terminal  have  not  been  worked 
out,  nor  have  the  all-important  procedures  for  invoking  relevant  tutorial  networks 
on-line.  Not  only  ease  of  use,  but  "pleasure  in  use"  is  essential  to  continued 
acceptance  of  the  interface,  and  of  the  procedures  implicit  in  the  interface. 

Another  key  factor  will  be  the  performance  requirements  of  the  machine  as  manifest 
in  its  delays  in  responding  to  conmands  and  requests  by  the  user,  especially  in 
data  searches  and  retrievals.  Delays  of  more  than  a very  few  seconds  can  radically 
alter  the  level  of  aspiration  the  user  adopts  for  task  performance;  hence,  for 
the  task  itself.  The  interface  design  is  therefore  a complex  of  system  tradeoff 
decisions  and  inventions.  These  require  formulation  and  testing. 

Training  of  Assessors.  It  remains  to  be  seen  whether  individuals  can  be  suitably 
selectea  and  trainecPin  the  knowledge  and  disciplines  of  the  assessor  role  in 
ETAM  procedures.  In  the  field  of  training,  as  in  others,  people  tend  to  terome 
specialists  and  prefer  the  well-identified  role  of  specialized  expertise.  The 
assessor  must  be  an  "operational  generalist,"  one  familiar  with  the  workings  of 
a broad  range  of  subject  matter  and  able  to  cope  with  the  complexity  of  diversity 
and  able  to  be  comfortable  with  ambiguity  and  imprecision  as  known  to  the  specialist. 
It  is  unclear  that  even  a very  small  number  of  such  individuals  can  be  found  and 
justified  as  well  as  qualified  for  this  job. 
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Acceptance  by  Decision-Makers.  The  underlying  rationales  in  ETAM  are  order, 
consistency  and  explicitness  in  decision  data.  Many  executives  may  feel  threaten- 
ed by  these  factors,  and  especially  the  last-named.  The  explicit  documentation 
may  conceivably  make  them  vulnerable  to  their  competitive  peers  or  to  their 
higher  command  levels.  The  high  degree  of  order  in  ETAM  data  and  its  formal 
retrieval  structures  may  shift  the  executive's  competence  from  mere  dependence 
on  the  output  of  trusted  staff  to  evaluation  of  constituent  judgments  by  his 
staff.  This,  in  turn,  may  imply  a higher  level  of  operating  knowledge  about  the 
workings  of  the  enterprise  than  the  executive  may  feel  he  has.  Any  one  or  all 
of  these  factors,  added  to  novelty  in  operation,  could  result  in  resistance  from 
those  whom  the  function  is  intended  to  serve. 

Training  Development  Innovations.  The  ETAM  Range-of-Effect  topics  included  oper- 
ations in  the  design  and  development  of  training  content:  description  of  system 

requirements,  task  requirements,  selection  of  instructional  devices,  development 
of  training  content,  and  so  on.  These  were  acknowledged  as  a special  class  of 
problems  for  decisions  about  innovations.  An  originally  proposed  ETAM  study  and 
development  task  was  targeted  at  the  Training  Development  and  Administrative 
functions.  This  was  not  undertaken  in  the  current  ETAM  effort.  It  would  have 
refined  the  evaluation  of  the  effects  of  an  innovation  upon  the  job/tasks  associ- 
ated with  the  spectrum  of  training  development  and  adminstrative  function.  Con- 
troversies about  "Instructional  Systems  Development"  give  evidence  that  these 
are  not  a trivial  class  of  problems  for  which  innovations  are  important.  This 
entire  topic  needs  to  be  brought  out  and  given  more  sophisticated  attention  in 
terms  of  benefit-cost  analysis. 

RECOMMENDATIONS 

ETAM  has  been  designed  procedurally  and  certain  components  are  computerized.  It 
still  requires  technical  nurturing  and,  perhaps  more  important  for  survival, 
promotion  for  acceptance  both  as  a concept  and  as  an  operational  entity.  It  is 
essential  that  it  be  integrated  as  an  operational  concept  into  the  decision- 
making process  for  assessing  innovation  and  change. 

It  is  recoronended  that  the  existing  Range-of-Effect  descriptors  be  carefully 
evaluated  using  training  and  training  research  professionals  who  are  familiar 
with  the  content  and  intent  of  training  programs,  and  who  have  thoroughly 
familiarized  themselves  with  the  taxonomic  structures  outlined  in  Appendix  A of 
this  report.  The  existing  sample  of  entities  in  the  Abbreviated  Data  Bases  can 
be  used  initially,  however,  familiarity  with  the  descriptors  can  best  be  gained 
by  indexing  and  storing  additional  items  in  the  data  base.  This  indexing  of 
entities  is  intended  to  have  Its  primary  value  in  the  ETAM  application,  however, 
the  structure  lends  Itself  to  additional  investigations  by  training  and  personnel 
researchers  into  the  profiles  of  courses,  instructional  vehicles,  and  job/tasks 
used  within  the  Navy.  This  understanding  should  lead  to  more  specific  efforts 
to  improve  these  functions. 

The  decision  analysis  and  financial  analysis  function  within  ETAM  provide  a 
strong  analytic  capability  for  assessment  of  Innovations.  The  decision  tree 
permits  outcome  utilities  to  be  evaluated  based  upon  a "maximization  of  expected 
value"  decision  rule.  This  would  appear  to  be  the  most  appropriate  rule  for  the 
majority  of  changes  Introduced  into  the  training  system.  However,  considering 
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the  Navy's  mission  and  the  possibility  of  major  risks  to  the  performance  of  that 
mission  by  Introducing  a significant  functional  change  to  training;  e.g.,  <con- 
verslon  of  a high  number  of  technical  courses  to  OJT,  other  decision  ru'les  such 
as  "minimization  of  expected  loss"  should  be  considered.  Scenarios  should  be 
developed  and  tested  using  the  presently  designed  assessment  procedures. 

Another  aspect  of  the  decision  tree  structure  that  should  be  given  further  study 
is  the  means  by  which  risk  reduction  projects  are  selected  and  grouped.  Optimi- 
zation techniques  may  be  applicable  for  deciding  on  the  most  appropriate  groupings 
for  maximizing  the  decision  variable. 

INTRODUCTION  OF  CHANGE  AGENT.  The  single  most  important  action  that  the  sponsor- 
ing agency  could  take  would  be  the  appointment  of  an  activist  change  agent  whose 
roles  and  responsibilities  would  be  much  like  those  described  for  the  change 
agent  in  Chapter  XIII  of  the  Major  Innovations  in  Training  Technology  report. 

Its  future  stages  of  maturation  will  depend  not  on  sponsoring  agency  relation- 
ships  with  designers  and  developers,  but  on  personnel  roles  and  responsibilities 
within  the  Navy.  The  individual  or  team  selected  should  have  proprietary  moti- 
vations and  Incentives. 

The  change  agent  should  reside  in  an  organization  that  gives  him  mobility  and 
permanence.  He  needs  mobility  for  access  to  the  variety  of  Navy  agencies  that 
generate  data  base  input  to  ETAM,  and  to  agencies  that  can  be  guided  b.y  various 
ETAM  functions.  The  change  agent  can  then  serve  the  advisory  roles  so  important 
to  the  promotion  of  even  limited  adoptions.  And  the  network  of  organizational 
dependencies  can  be  strengthened  as  well  as  articulated. 

Promotional  skills  and  initiatives  must  be  supplemented  with  sound  technical 
expertise,  and  both  of  these  tempered  with  optimistic  realism.  The  technical 
expertise  includes  a grasp  of  ETAM  principles  and  philosophies  as  well  as  its 
practices  and  extends  into  Navy  training  operations  and  its  realities,  both 
technical  and  organizational. 

PAPER  WALK-THROUGHS  OF  ETAM.  Two  or  three  samples  of  real  problems  concerning 
decisions  about  long-term  changes  in  Navy  training  policy  or  practice  or  equip- 
ment should  be  selected  for  trial  runs  of  ETAM.  Several  individuals,  at  least 
tentatively  supportive  of  ETAM  concepts,  on  the  sponsoring  agency's  staff  should 
conduct  a walk-through  simulation  of  the  eight  steps  in  the  procedure.  The 
results  should  be  far  more  convincing  than  sample  exercises  conducted  by  the 
design  team.  The  value  of  such  extensions  of  present  ETAM  status  as  range-of- 
effect  indexing  and  fully  interactive  supports  can  then  be  reassessed  with  an 
empirical  reference. 

GRAPHIC  DISPLAY  TERMINALS.  Ultimately,  a tool  such  as  ETAM  should  interface 
with  the  assessor  through  a graphic  CRT  type  terminal,  rather  than  the  existing 
printer  terminal.  This  is  seen  as  the  key  to  successful  interaction  between 
assessors,  executives  and  ETAM  procedures.  Such  a terminal  makes  feasible  the 
use  of  the  computer  as  a training  and  guidance  capability,  and  an  effective 
basis  for  working  with  data  base  content.  The  thirty  characters  per  second 
output,  limited  to  alphamerlcs,  of  the  terminal  station  is  too  limited  for  much 
more  than  limited  demonstrations  of  feasibility.  The  light  pen  or  cursor  coupled 
with  dynamic  menus  of  commands  and  instant  tutorials  and  guidance  messages  would 
dramatically  change  the  psychological  nature  of  the  procedures.  Ease  of  use, 
flexibility  and  convenience  would  increase,  perhaps  by  an  order  of  magnitude. 
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The  development  of  these  interactive  capabilities  could  be  made  to  serve  purposes 
broader  than  those  of  formal  ETAM.  Cost  analysis  and  sensitivity  analysis 
(Tasks  4 and  6 in  ETAM)  have  a broader  application  base.  So  does  the  financial 
analysis  capability  (Task  7). 

If  this  recommendation  is  followed,  the  design  and  development  of  the  interactive 
capability  should  have  a strong  control  from  human  factors  expertise  on  the 
programming  effort. 

INTERACTIVE  ETAM  TRAINING  PROGRAM.  The  content  for  training  and  guiding  the 
ETAM  assessor  staff  and  executive  users  already  exists  in  the  text  of  ETAM 
documents--the  Phase  1 1 -A  report  and  the  reports  on  scaling  and  indexing,  and 
the  innovations  in  training  report.  The  content  of  these  documents  is  also 
appropriately  indexed  according  to  steps  and  considerations  in  the  ETAM  procedures. 
The  availability  of  a graphic  display  terminal  is  seen  as  a means  to  facilitate 
a "learn  by  doing"  approach  to  improving  the  assessors'  ability  to  interact  with 
the  ETAM  procedures.  The  speed  of  the  present  printer  terminal  precludes  effec- 
tive application  of  training  content  integrated  into  the  ETAM  computerized  process. 
The  availability  of  a higher  speed  graphic  display  terminal  would  permit  the  train- 
ing in  ETAM  to  be  threaded  on  sample  problems  with  realistic  contents. 

It  is  also  tentatively  suggested  that  the  ETAM  training  material  might  be  used 
more  widely.  It  could  provide  a general  working  knowledge  of  instructional 
technology  relatively  painlessly  to  a wide  spectrum  of  specialists  and  managers 
in  the  Navy’s  training  and  education  operations.  Hopefully,  this  enlightenment 
could  not  only  widen  the  perspective  of  specialists  but  also  increase  the  en- 
lightened cooperation  among  agencies  that  tend  to  develop  rivalries. 

LINKED  NAVY  DATA  BASES.  The  range-of-effect  task  and  the  cost  and  financial 
analysis  tasks  in  ETAM  point  to  the  profound  need  for  shared  disciplines  in 
accessing  the  content  from  large  Navy  data  bases  that  are  scattered  among  differ- 
ent organizations.  The  key  to  making  these  data  bases  into  problem-solving 
networks  is  a categorical  discipl ine--the  selection  and  meaning  given  to  the 
terms  that  index  data  elements.  Only  by  a shared  taxonomic  discipline  can  one 
data  base  "talk"  to  another,  and  share  data  constituents  in  arriving  at  non- 
local decisions. 

The  need  for  multipurpose  data  comes  to  a focus  in  ETAM  determinations  of  range- 
of-effect  and  in  cost  analysis.  In  most  cases,  human  judgment  (hopefully  informed) 
is  the  link  between  the  ETAM  problem  and  external  data  bases  on  courses,  student 
attributes,  job-tasks,  cost  breakdowns,  and  instructional  equipment.  This  link 
can  be  expensive,  time-consuming  and,  contingent  in  part  on  imprecise  or  ambiguous 
source  data,  unreliable. 

Two  courses  are  open.  One  is  to  determine  what  is  the  best  that  can  be  cone 
with  existing  data  sources  and  structures.  The  other  is  to  attempt  at  least 
compromise  improvements  in  getting  the  source  to  better  match  the  need.  These 
are  not,  of  course,  mutually  exclusive  alternatives.  A single  agency  is  unlikely 
to  be  able  to  directly  affect  data  base  disciplines  in  a broad  span  of  organi- 
zations. But  if  ETAM  becomes  adopted  as  a rational  technique  for  making  decisions 
about  any  of  a broad  range  of  change  in  policy  and  practice  of  Navy  training,  it 
can  become  a common  denominator  definition  of  data  support  requirements.  It  may 
therefore  serve  as  one  major  reference  for  data  base  standardizations. 
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ETAM  was  designed  to  accept  and  use  whatever  data  come  In  to  It.  Including  gross 
estimates  as  well  as  guesses.  This  Is  both  a power  and  a liability.  The  power 
is  In  getting  a job  done,  a decision  made;  even  though  the  source  data  arc  weak, 
the  best  that  can  be  made  of  weak  data  Is  done  with  it.  It  has  built-in  expressions 
for  weakness  In  dataconfldence  ranges  and  limits  expressed  by  the  assessor  to 
the  decision  maker  In  several  ways.  But  the  liability  Is  that  despite  expressions 
of  low  confidence  In  source  data,  the  formalism  Implied  by  "models"  and  "computer- 
ization" can  create  an  atmosphere  of  spurious  rightness  of  content.  This  liability 
Is  not  peculiar  to  ETAM— It  applies  to  any  form  of  computerization.  The  awareness 
merely  Increases  the  ethical  responsibility  for  system  designers  to  strive  for 
legitimate  information  Input. 

This  Is  a general  comment  without  a specific  focus.  Ambiguity  and  uncertainty 
are  Inevitable  In  all  real  life  operations,  and  It  Is  unrealistic  to  expect 
otherwise.  Decisions  will  continue  to  be  made  under  massive  uncertainties. 

ETAM  has  attempted  to  reduce  uncertainty  partly  by  structuring  It  through  showing 
what  variables  are  relevant  to  the  decision,  partly  by  processing  whatever  data 
can  be  expressed  towards  solutions.  Further  Improvements  In  the  quality  of 
decisions  will  have  to  come  from  the  quality  of  Input  data. 

FUTURE  DEVELOPMENTS.  The  content  of  ETAM  Is  specific  to  education  and  training 
technology  and  operations.  The  structure,  as  represented  by  the  eight  procedural 
tasks  and  their  constituent  procedures,  Is  a completely  general  one  for  arriving 
at  and  making  decisions  that  affect  changes  In  an  enterprise  and  what  It  does. 

It  Is  anticipated  that  this  generality  of  procedural  structure  for  arriving  at 
rational  decisions— even  In  non-ratlona!  contexts— can  be  recognized  and  translated 
widely  Into  other  technical  and  organizational  environments.  It  Is  recognized 
that  many  organizations  now  use  cost  models  and  financial  planning  models;  a few 
even  apply  some  form  of  utility  scaling  In  formulating  decision  models.  But  It 
Is  suspected  that  what  In  ETAM  are  Tasks  1 through  5 are  not  tackled  formally, 
so  that  the  latter  tasks  work  from  limited  and  often  unexplored  data  sources 
that  are  essential  to  sound  decision  making.  Nor  Is  the  decision  maker  enabled 
quickly— such  as  within  the  hour's  presentation  and  study— to  sample  selectively 
Into  the  background  data  and  Judgments  leading  to  the  final  chart  of  "conclusions." 
It  Is  the  limited  time  and  span  of  attention  of  the  executive  who  makes  the 
choice  of  commitment  that  largely  Justifies  the  quality  of  the  assessing  staff 
work. 

If  these  anticipations  come  about,  ETAM  Itself  will  have  served  a change  agent 
function. 
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